Python & XML
Christopher A. Jones
Fred L. Drake, Jr.
Publisher: O'Reilly
First Edition January 2002 ISBN: 0-596-00128-2, 384 pages
Python is an ideal language for manipulating XML, and this new volume gives you a solid foundation for using these two languages together. Complete with practical examples that highlight common application tasks, the book starts with the basics then quickly progresses to complex topics like transforming XML with XSLT and querying XML with XPath. It also explores more advanced subjects, such as SOAP and distributed web services.
Copyright © 2002 O'Reilly & Associates, Inc. All rights reserved.
Printed in the United States of America.
Published by O'Reilly & Associates, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O'Reilly & Associates books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safari.oreilly.com). For more information contact our corporate/institutional sales department: 800-998-9938 or [email protected].
Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of O'Reilly & Associates, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. The association between the image of elephant shrews and Python and XML is a trademark of O'Reilly & Associates, Inc.
While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.
IT-SC book 1
Dedication
Preface Audience Organization Conventions Used in This Book How to Contact Us Acknowledgments
1. Python and XML 1.1 Key Advantages of XML 1.2 The XML Specifications 1.3 The Power of Python and XML 1.4 What Can We Do with It?
2. XML Fundamentals 2.1 XML Structure in a Nutshell 2.2 Document Types and Schemas 2.3 Types of Conformance 2.4 Physical Structures 2.5 Constructing XML Documents 2.6 Document Type Definitions 2.7 Canonical XML 2.8 Going Beyond the XML Specification
3. The Simple API for XML 3.1 The Birth of SAX 3.2 Understanding SAX 3.3 Reading an Article 3.4 Searching File Information 3.5 Building an Image Index 3.6 Converting XML to HTML 3.7 Advanced Parser Factory Usage 3.8 Native Parser Interfaces
4. The Document Object Model 4.1 The DOM Specifications 4.2 Understanding the DOM 4.3 Python DOM Offerings 4.4 Retrieving Information 4.5 Changing Documents 4.6 Building a Web Application 4.7 Going Beyond SAX and DOM
5. Querying XML with XPath 5.1 XPath at a Glance 5.2 Where Is XPath Used? 5.3 Location Paths 5.4 XPath Arithmetic Operators 5.5 XPath Functions 5.6 Compiling XPath Expressions
IT-SC book 2 6. Transforming XML with XSLT 6.1 The XSLT Specification 6.2 XSLT Processors 6.3 Defining Stylesheets 6.4 Using XSLT from the Command Line 6.5 XSLT Elements 6.6 A More Complex Example 6.7 Embedding XSLT Transformations in Python 6.8 Choosing a Technique
7. XML Validation and Dialects 7.1 Working with DTDs 7.2 Validation at Runtime 7.3 The BillSummary Example 7.4 Dialects, Frameworks, and Workflow 7.5 What Does ebXML Offer?
8. Python Internet APIs 8.1 Connecting Web Sites 8.2 Working with URLs 8.3 Opening URLs 8.4 Connecting with HTTP 8.5 Using the Server Classes
9. Python, Web Services, and SOAP 9.1 Python Web Services Support 9.2 The Emerging SOAP Standard 9.3 Python SOAP Options 9.4 Example SOAP Server and Client 9.5 What About XML-RPC?
10. Python and Distributed Systems Design 10.1 Sample Application and Flow Analysis 10.2 Understanding the Scope 10.3 Building the Database 10.4 Building the Profiles Access Class 10.5 Creating an XML Data Store 10.6 The XML Switch 10.7 Running the XML Switch 10.8 A Web Application
A. Installing Python and XML Tools A.1 Installing Python A.2 Installing PyXML A.3 Installing 4Suite
B. XML Definitions B.1 XML Definitions
C. Python SAX API
D. Python DOM API D.1 4DOM Extensions
IT-SC book 3
E. Working with MSXML3.0 E.1 Setting Up MSXML3.0 E.2 Basic DOM Operations E.3 MSXML3.0 Support for XSLT E.4 Handling Parsing Errors E.5 MSXML3.0 Reference
F. Additional Python XML Tools F.1 Pyxie F.2 Python XML Tools F.3 XML Schema Validator F.4 Sab-pyth F.5 Redfoot F.6 XML Components for Zope F.7 Online Resources
Colophon
IT-SC book 4 Dedication
We would like to dedicate this book to Frank Willison, O'Reilly Editor- in-Chief and Python Champion
——Christopher A. Jones and Fred L. Drake, Jr.
Frank will be remembered in the Python community for the several great Python books that he made possible, memories of his participation in many Python conferences, and his Frankly Speaking columns. The Python world (and the world at large) won't be the same without Frank.
——Guido van Rossum, Python creator
IT-SC book 5 Preface
This book comes to you as a result of the collaboration of two authors who became interested in the topic in very different ways. Hopefully our motivations will help you understand what we each bring to the book, and perhaps prove to be at least a little entertaining as well.
Chris Jones started using XML several years ago, and began using Python more recently. As a consultant for major companies in the Seattle area, he first used XML as the core data format for web site content in a home-grown publishing system in 1997. But he really became an XML devotee when developing an open source engine, which eventually became the key technology for Planet 7 Technologies. As a consultant, he continues to use XML on an almost daily basis for everything from configuration files to document formats.
Chris began dabbling in Python because he thought it was a clean, object-oriented alternative to Perl. A long-time Unix user (but one who frequently finds himself working with Windows in Seattle), he has grown accustomed to scripting languages that place the full Unix API in the hands of developers. Having used far too much Java and ASP in web development over the years, he found Python a refreshing way to keep object-orientation while still accessing Unix sockets and threads—all with the convenience of a scripting language.
The combination of Python and XML brings great power to the developer. While XML is a potent technology, it requires the programmer to use objects, interfaces, and strings. Python does so as well, and therefore provides an excellent playpen for XML development. The number of XML tools for Python is growing all the time, and Chris can produce an XML solution in far less time using Python than he can with Java or C++. Of course, the cross-platform nature of Python keeps our work consistently usable whether we're developing on Windows, Linux, or a Unix variant—the combination of which we both seem to find powerful.
Fred Drake came to Python and XML from a different avenue, arriving at Python before XML. He discovered Python while in graduate school experimenting with a number of programming languages. After recognizing Python as an excellent language for rapid development, he convinced his advisors that he should be able to write his masters project using Python. In the course of developing the project, he became increasingly interested in the Python community. He then made his first contributions to the Python standard library, and in so doing became noticed by a group of Python programmers working on distributed systems projects at the research organization of CNRI. The group was led by Guido van Rossum, the creator of Python. Fred joined the team and learned more about distributed systems and gluing systems together than he ever expected possible, and he loved it.
While still in graduate school, Fred argued that Python's documentation should be converted to a more structured language called SGML. After a few years at CNRI, he began to do just that, and was able to sink his teeth into the documentation more vigorously. The SGML migration path eventually changed to an XML migration path as XML acceptance grew. Though that goal has not yet been achieved (he is still working on it), Fred has substantially changed the way the documentation is maintained, and it now represents one of the most structured applications of the typesetting and document markup system developed by Donald Knuth and Leslie Lamport.
Over time, the team from CNRI became increasingly focused on the development of Python, and moved on to form PythonLabs. Fred remained active in XML initiatives around Python and
IT-SC book 6 pushed to add XML support to the standard library. Once this was achieved, he returned to the task of migrating the Python documentation to XML, and hopes to complete this project soon.
Audience
This book is for anyone interested in learning about using Python to build XML applications. The bulk of the material is suited for programmers interested in using XML as a data interchange format or as a transformable format for web content, but the first half of the book is also useful to those interested in building more document-oriented applications.
We do not assume that you know anything about XML, but we do assume that you have looked at Python enough that you are comfortable reading straightforward Python code; however, you do not need to be a Python guru. If you do not know at least a little Python, please consult one of the many excellent books that introduce the language, such as Learning Python, by Mark Lutz and David Ascher and Lutz (O'Reilly, 1999). For the sections where web applications are developed, it helps to be familiar with general concepts related to web operations, such as HTTP and HTML forms, but sufficient information is included to get you started with basic CGI scripting.
Organization
This book is divided into ten chapters and six appendixes, as follows:
Chapter 1
This chapter offers a broad overview of XML and why Python is particularly well-suited to XML processing.
Chapter 2
This chapter provides a good introduction to XML for newcomers and a refresher for programmers who have some familiarity with the standard.
Chapter 3
This chapter gives a detailed introduction to using Python with the SAX interface, for generating parse events from an XML data stream.
Chapter 4
This chapter provides an introduction to working with DOM, which is the dominant object-oriented, tree-based API to an XML document.
Chapter 5
This chapter discusses using a traversal language to extract portions of documents that meet your application's requirements.
Chapter 6
This chapter details using XSLT to perform transformations on XML documents.
IT-SC book 7 Chapter 7
This chapter discusses validating XML generated from other sources.
Chapter 8
This chapter provides an overview of Python's high-level support for Internet protocols, including tools for building both clients and servers for HTTP.
Chapter 9
This chapter offers discussion of and examples showing how to build and use web services with Python.
Chapter 10
This chapter is an extended example that shows a variety of approaches to applying Python in constructing an XML-based distributed system.
Appendix A
This appendix provides instructions on installing Python and the major XML packages used throughout this book.
Appendix B
This appendix gives a list of definitions from the XML specification and a Python script to extract them from the specification itself.
Appendix C
This appendix offers detailed API information for using the dominant event- based XML interface in Python.
Appendix D
This appendix provides detailed interface documentation for using the standard tree-oriented API for XML from Python.
Appendix E
This appendix gives information on Microsoft's XML libraries available for Python.
Appendix F
This appendix is a summary of the many additional tools that are available for using XML with Python, and a list of starting points for additional information on the Web.
Conventions Used in This Book
IT-SC book 8 The following typographical conventions are used throughout this book:
Bold
Used for the occasional reference to labels in graphical user interfaces, as well as user input.
Italic
Used for commands, URLs, filenames, file extensions, directory or folder names, emphasis, and new terms where they are defined.
Constant width
Used for constructs from programming languages, HTML, and XML, both within running text and in listings.
Constant width italic
Used for general placeholders that indicate that an item should be replaced by some actual value in your own program. Most importantly, this font is used for formal parameters when discussing the signatures of API methods.
How to Contact Us
We have tested and verified all the information in this book to the best of our abilities, but you may find that features have changed or that we have let errors slip through the production of the book. Please let us know of any errors that you find, as well as suggestions for future editions, by writing to:
O'Reilly & Associates, Inc.
1005 Gravenstein Highway North
Sebastopol, CA 95472
1-800-998-9938 (in the United States or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (fax)
You can also send us messages electronically. To be put on the mailing list or to request a catalog, send email to: [email protected]
To ask technical questions or comment on the book, send email to: [email protected]
IT-SC book 9 We have a web site for the book, where we'll list examples, errata, and any plans for future editions. You can access this page at: http://www.oreilly.com/catalog/pythonxml/
For more information about this book and others, see the O'Reilly web site: http://www.oreilly.com/
IT-SC book 10 Acknowledgments
While it is impossible to individually acknowledge everyone that had a hand in getting this book from an idea to the printed work you now hold in your hand, we would like to recognize and thank a few of these special people.
We are both very grateful for the support of our families, without which this would not have even gotten started. Chris would like to thank his family (Barb, Miles, and Katherine); without their support he would never get any writing completed, ever. Fred owes a great deal of gratitude to his wife (Cathy), who spent many a lonely evening wondering if he'd remember to come to bed. His children (William, Christopher, and Erin) made sure he didn't forget why he spends so much time on all this. Those late-night trips to the coffee shop with Erin will never be forgotten!
We'd especially like to thank Guido van Rossum and Fred's compatriots at PythonLabs (Tim Peters, Jeremy Hylton, and Barry Warsaw) for making sure Python could grow to be such a wonderful tool for building applications, and for leading the incredible community efforts which have gone into both Python itself and the excellent selection of additional packages of Python code.
Python's development has been beleaguered by regular employment changes, but we all owe a debt of gratitude to the employers of the contributors and the PythonLabs team. Now at Zope Corporation (formerly Digital Creations), PythonLabs has finally found a home that offers both a rich environment for Python and comfortable place to settle down. Previous employers of Python's lead developers, including the Corporation for National Research Initiatives (CNRI) and Stichting Mathematisch Centrum, deserve credit for allowing Python to germinate and blossom.
Our reviewers' efforts were invaluable and made this book what it is today. (They were helpful, and showed great faith in our ability to pull this off, even when we weren't so sure.) Martin von Löwis, Paul Prescod, Simon St.Laurent, Greg Wilson, and Frank Willison all contributed generously of their time and helped to ensure that our mistakes were noticed. The feedback they provided, both from a development and from a technical support perspective, was invaluable. Any mistakes in the finished book are our own. Fred Drake, who began working on this project as a technical reviewer, must still answer for any mistakes he's introduced!
Many people at O'Reilly played an important part in the development of this book, and without the help of their editorial staff, this book would seem rambling and incoherent (well, more so at least!). Laura Lewin deserves special recognition. Without her editorial skill and faith in our ability to present the important aspects of our subject, you wouldn't be reading this; her penchant for reminding us of the big picture when we became mired in the particulars of topics kept us on track and focused. Frank Willison deserves a great deal of credit not only for bringing Laura to O'Reilly, but in shepherding O'Reilly's efforts to bring together their line of books on Python; we'll all miss him. Finally, we'd like to thank the production staff at O'Reilly for their hard work in getting the book to print.
IT-SC book 11 Chapter 1. Python and XML
Python and XML are two very different animals, each with a rich history. Python is a full-scale programming language that has grown from scripting world roots in a very organic way, through the vision and guidance of Python's inventor, Guido van Rossum. Guido continues to take into account the needs of Python developers as Python matures. XML, on the other hand, though strongly impacted by the ideas of a small cadre of visionaries, has grown from standards- committee roots. It has seen both quiet adoption and wrenching battles over its future. Why bother putting the two technologies together?
Before the Python/XML combination, there seemed no easy or effective way to work with XML in a distributed environment. Developers were forced to rely on a variety of tools used in awkward combination with one other. We used shell scripting and Perl to process text and interact with the operating system, and then used Java XML API's for processing XML and network programming. The shell provided an excellent means of file manipulation and interaction with the Unix system, and Perl was a good choice for simple text manipulation, providing access to the Unix APIs. Unfortunately, neither sported a sophisticated object model. Java, on the other hand, featured an object-oriented environment, a robust platform API for network programming, threads, and graphical user interface (GUI) application development. But with Java, we found an immediate lack of text manipulation power; scripting languages typically provided strong text processing. Python presented a perfect solution, as it combines the strengths of all of these various options.
Like most scripting languages, Python features excellent text and file manipulation capabilities. Yet, unlike most scripting languages, Python sports a powerful object-oriented environment with a robust platform API for network programming, threads, and graphical user interface development. It can be extended with components written in C and C++ with ease, allowing it to be connected to most existing libraries. To top it off, Python has been shown to be more portable than other popular interpreted languages, running comfortably on platforms ranging from massive parallel Connection Machines to personal digital assistants and other embedded systems. As a result, Python is an excellent choice for XML programming and distributed application development.
It could be said that Python brings sanity and robustness to the scripting world, much in the same way that Java once did to the C++ world. As always, there are trade-offs. In moving from C++ to Java, you find a simpler language with stronger object-oriented underpinnings. Changing to a simpler language further removed from the low-level details of memory management and the hardware, you gain robustness and an improved ability to locate coding errors. You also encounter a rich API equipped with easy thread management, network programming, and support for Internet technologies and protocols. As may be expected, this flexibility comes at a cost: you also encounter some reduced performance when comparing it with languages such as C and C++.
Likewise, when choosing a scripting language such as Python over C, C++, or even Java, you do make some concessions. You trade performance for robustness and for the ability to develop more rapidly. In the area of enterprise and Internet systems development, choosing reliable software, flexible design, and rapid growth and deployment are factors that outweigh the performance gains you might get by using a language such as C++. If you do need some of the performance back, you can still implement speed-sensitive components of your application in C or C++, but you can avoid doing so until you have profiling data to help you pinpoint what is
IT-SC book 12 really a problem and what only might be a problem. (How to perform the analysis and write extensions in C/C++ is a topic for other books.)
Regardless of your feelings on scripting languages, Java, or C++, this book focuses on XML and the Python language. For those who are new to XML, we will start with an overview of why it is interesting, and then we'll move on to using it from Python and seeing how we make our XML applications easier to create.
1.1 Key Advantages of XML
XML has a few key advantages that make it the data language of choice on the Internet. These advantages were designed into XML from the beginning, and, in fact, are what make it so appealing to Internet developers.
1.1.1 Application Neutrality
First, XML is both human- and machine-readable. This is not a subtle point. Have you ever tried to read a Microsoft Word document with a text editor? You can't if it was saved as a .doc file, because the information in a .doc document is in a binary (computer readable only) format, even though most Word documents primarily consist of text. A Word document cannot be shared with any other application besides Word—unless that application has been taught the intricacies of Word's binary format. In this case, the application must also be taught to expect changes in Word's format each time there is a new release from Microsoft.
This sounds annoying for the developer, but how bad is it, really? After all, Word is incredibly popular, so it must not be too hard to figure out. Let's look at the top of the Word file that contains this chapter:
Ï_ࡱ_á > _ ÿ _ _ B_ _ D_ _
ÿÿÿ ?_ @_ A_ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿì¥Á 7 _ _¿ _ _ >_ _ bjbjU_U_ __ 0¸_ 7| 7| W_ _ C
ÿÿ_ ÿÿ_ ÿÿ_ l Ê_
Ê_ Ê_ Ê_ Ê_ Ê_ Ê_ ¶ _
This certainly looks familiar to anyone who has ever opened a Word file with a text editor. We don't see our recognizable text (the content we intended) so we must assume it is buried deep in
IT-SC book 13 the file. Determining what the true content is and where it is can be difficult, but it shouldn't be. It is our data, after all. Let's try another supported format: "Rich Text Format," or RTF. Unlike the .doc file, this format is text-based, and should therefore be a bit easier to decipher. We search down in the file to find the start of our text:
\par }\pard \s34\qr
\li0\ri0\sb80\sa480\sl240\slmult0\widctlpar\aspalpha\aspnum\faauto\out linelevel0\widctlpar\aspalpha\aspnum\faauto\outlinelevel0\pnrauth1\pnr date-967302179\pnrnot1\adjustright\rin0\lin0\itap0 {\b0\fs48 Combining
Python and XML}{
\b0\deleted\fs48\revauthdel1\revdttmdel-2041034726 Fundamentals}{\b0\f s48\revised\revauth1\revdttm-2041034726 ?}{\b0\fs48
\par }\pard\plain \qj
This is better. The chapter title is visible, so we can try to decipher the structure from that point forward. The markup appears to be complex, and there's a hint of an old version of the chapter title. To extract the text we actually want, we need to understand the Word model for revision tracking, which still presents many challenges.
XML, on the other hand, is application-neutral. In other words, an XML document is usually processed by an XML parser or processor, but if one is not available, an XML document can be easily read and parsed. Data kept in XML is not trapped within the constraints of one particular software application. The ability to read rich data files can become very valuable when, for example, 20 years from now, you dig up a CD-ROM of old business forms that you suddenly find you need again. Will QuickBooks still allow you to extract this same data in 2021? With XML, you can read the data with any text editor.
Let's look at this chapter in XML. Using markup from a common document type for software manuals and documentation (DocBook), it appears somewhat verbose, and doesn't include change-tracking information, but we can identify the text quite easily now:
rich history. Python is a full-scale programming language that has grown
from scripting world roots, and has done so in a very organic way
Note that additional characters appear in the document (other than the document content); these are called markup (or tags). We saw this in the RTF version of the document as well, but there were many more bits of text that were difficult to decipher, and we can reasonably surmise that the strange data in the MS Word document would correspond to this in some way. Were this a
IT-SC book 14 book on RTF, you would quickly surmise two things: RTF is much more like a printer control language than the example of XML we just looked at, and writing a program that understands RTF would be quite difficult. In this book, we're going to show you that XML can be used to define languages that fit your application, and that creating programs that can decipher XML is not a difficult task, especially with the help of Python.
1.1.2 Hierarchical Structure
XML is hierarchical, and allows you to choose your own tag names. This is quite different from HTML. In XML, you are free to create elements of any type, and stack other elements within those elements. For example, consider an address entry:
In the above well-formed XML code, I came up with a few record names and then lumped them together with data. XML processing software, such as a parser (which you use to interpret the syntactic constructs in an XML document), would be able to represent this data in many ways, because its structure has been communicated. For example, if we were to look at what an application programmer might write in source code, we could turn this record into an object initialized this way: addr = Address( ) addr.name = "Bubba McBubba" addr.street = "123 Happy Go Lucky Ln." addr.city = "Seattle" addr.state = "WA" addr.zip = "98056"
This approach makes XML well-suited as a format for many serialized objects. (There are some constructs for which XML is not so well suited, including many formats for large numerical datasets used in scientific computing.) XML's hierarchical structure makes it easy to apply the concept of object interfaces to documents—it's quite simple to build application-specific objects directly from the information stream, given mappings from element names to object types. We later see that we can model more than simple hierarchical structures with XML.
1.1.3 Platform Neutrality
IT-SC book 15 Remember that XML is cross-platform. While this is mainly a feature of its text-based format, it's still very much true. The use of certain text encodings ensures that there are no misconceptions among platforms as to the arrangement of an XML document. Therefore, it's easy to pass an XML purchase order from a Unix machine to a wireless personal digital assistant. XML is designed for use in conjunction with existing Internet infrastructure using HTTP, SSL, and other messaging protocols as they evolve. These qualities make XML lend itself to distributed applications; it has been successfully used as a foundation for message queuing systems, instant messaging applications, and remote procedure call frameworks. We examine these applications further in Chapter 9 and Chapter 10. It also means that the document example given earlier is more than simply application-neutral, and can be readily moved from one type of machine to another without loss of information. A chapter of a technical book can be written by a programmer on his or her favorite flavor of Unix, and then sent to a publisher using book composition software on a Macintosh. The many difficult format conversions can be avoided.
1.1.4 International Language Support
As the Internet becomes increasingly pervasive in our daily lives, we become more aware of the world around us — it is a culture-rich and diversified place. As technologists, however, we are still learning the significance of making our software work in ways that supports more than one language at a time; making our text-processing routines "8-bit safe" is not only no longer sufficient, it's no longer even close.
Standards bodies all over the world have come up with ways that computers can interchange text written in their national languages, and sometimes they've come up with several, each having varying degrees of acceptance. Unfortunately, most applications do not include information about which language or interchange standard their data is written in, so it is difficult to share information across the cultural and linguistic boundaries the different standards represent. Sometimes it is difficult to share information within such boundaries if multiple standards are prominent.
The difficulties are compounded by very substantial cultural differences that present themselves about how text is handled. There are many different writing systems in addition to the western European left-to-right, top-to-bottom style in which this book is written; right-to-left is not uncommon, and top-to-bottom "lines" of text arranged right-to-left on the page is used in China. Hebrew uses a right-to-left writing system, but numbers are written using Arabic numerals from left to right. Other systems support textual annotations written in parallel with the text. Consider what happens when a document includes text from different writing systems!
Standards bodies are aware of this problem, and have been working on solutions for years. The editors of the XML specification have wisely avoided proposing new solutions to most of these issues, and are instead choosing to build on the work of experts on the topic and existing standards.
The International Organization for Standardization (ISO) and the Unicode Consortium (http://www.unicode.org/ ) have arrived at a single standard that, while not perfect, is perhaps the most capable standard attempting to unify the world's text representations, with the intent that all languages and alphabets (including ideographic and hieroglyphic character sets) are representable. The standard is known as ISO/IEC 10646, or more commonly, Unicode. Not all national standards bodies have agreed that Unicode is the standard for all future text interchange applications, especially in Asia, but there is widespread belief that Unicode is the best thing available to serve everyone. The standard deals with issues including multidirectional text,
IT-SC book 16 capitalization rules, and encoding algorithms that can be used to ensure various properties of data streams. The standard does not deal specifically with language issues that are not tied intimately to character issues. Software sensitive to natural language may still need to do a lot beyond using Unicode to ensure proper collation of names in a particular language (or multiple languages!). Some languages will require substantial additional support for proper text rendering (Arabic, for instance, which requires different letterforms for characters based on their position within a word and based on neighboring letterforms).
The World Wide Web Consortium (W3C) made a simple and masterful stroke to make it easier to use both the older interchange standards and Unicode. It required that all XML documents be Unicode, and specified that they must describe their own encoding in such a way that all XML processors were able to determine what encoding the document was written in. A few specific encodings must be recognized by all processors, so that it is always possible to generate XML that can be read anywhere and represent all of the world's characters. There is also a feature that allows the content of XML documents to be labeled with the actual language it is written in, but that's not used as much as it could be at this time.
Since XML documents are Unicode documents, the languages of the world are supported. The use of Unicode and encodings in XML are discussed in some detail in Chapter 2. Unicode strings have been a part of Python since Version 2.0, and the Python standard library includes support for a large number of encodings.
1.2 The XML Specifications
In the trade press, we often see references about how XML "now supports" some particular industry-specific application. The article that follows is often confused, offering some small morsel of information about an industry consortium that has released a new specification for an XML-based language to support interoperability of data within the consortium's industry. As technical people, we usually note that it doesn't apply to the industries we're involved in, or else it does, but the specification is too early a draft to be useful. In fact, our managers will probably agree with us most of the time, or they'll be privy to some relevant information that causes them to disagree. If we step up the corporate ladder a couple more rungs, however, we often find an increase in the level of confusion over XML. Sometimes, this is accompanied by either a call to "adopt XML" (too often with a list of particular specifications that are not intended to be used together), or a reaction that XML is too immature to use at all.
So we need to think about just what we can work with that will meet the following criteria:
It must make technical sense for our application.
It should be sufficiently well-defined that implementation is possible.
It must be able to be explained and justified to (at least) our direct managers.
It won't freak out the upper management.
Ok, we're technical people, so we may have to ignore that last item; it certainly won't be covered in this book. In fact, most of this really can't be covered in technical material. There are many specifications in various stages of maturity, and most are specific to one industry or another. However, we can point out what the foundation specifications are, because those you will need regardless of your industry or other requirements.
IT-SC book 17 1.2.1 XML 1.0 Recommendation
The XML specification itself is a document created and maintained by the W3C. As of this writing, the current version is Extensible Markup Language (XML) 1.0 (Second Edition), and is available from the W3C web site at http://www.w3.org/TR/REC-xml. (The second edition differs from the first only in that some editorial corrections and clarifications have been made; the specification is stable.)
XML itself is not a markup language, but a meta-language that can be used to define specific markup languages. In this, it inherits much from SGML. The specification covers five aspects of markup languages:
Range of structural forms which can be marked
Specific syntax of markup components
A schema language used to define specific languages
Definition of validity constraints
Minimum requirements for processing tools
Unlike SGML, XML allows itself to be used without defining an explicit markup language in any formal way. Whether or not this is useful for your applications, it has greatly accelerated the acceptance of XML-based technologies in some developer communities. This can happen because of the lower cost of entrance to the XML space. It is possible to adopt XML without learning some of the more esoteric corners of the specification, and development prototypes can start using XML technologies without a lot of advance planning.
Chapter 2 presents the most widely used parts of the specification and goes into more depth on what are the most important items to most readers of this book. If any of the details are of particular interest to you, please spend some time reading relevant parts of the specification. While it is at times a bit convoluted, it is not generally a difficult specification to read.
1.2.2 Namespaces in XML
While the XML 1.0 recommendation defines specific syntactic aspects of XML and one way of creating document types, it does not discuss how to combine components from multiple document types. The Namespaces in XML recommendation, available at http://www.w3.org/TR/REC-xml-names (referred to as Namespaces from now on), deals with the syntactic and structural mechanics of combining structured components from different specifications, but is largely silent on the meaning of resulting combinations. For this, it defers to specifications that had not been written when Namespaces was published.
This recommendation places some additional constraints on the syntactic construction of conformant documents. It allows a document to specify the source of each element or attribute by placing it in a namespace. Each namespace provides definitions for elements and attributes. How the elements and attributes are defined is not covered in this specification, so the concept of validation of an arbitrary document that uses namespaces is not entirely clear. It is possible to create a document type using XML 1.0 that has some support for namespaces, but such a schema loses much of the flexibility offered by the Namespaces specification. For example, the document
IT-SC book 18 type would have to specify the particular prefixes to which each namespace is bound, while the Namespaces specification allows prefixes to be determined by the document rather than the schema. Alternate schema languages that have better support for Namespaces have been defined; these are discussed briefly in Chapter 2.
1.2.3 XML as a Foundation
Like its predecessor SGML, XML provides a way to define languages that fit the requirements of your application. By specifying the exact syntax of the grammatical elements (such as the characters used to mark the start of an element), it has reduced the effort required to build conforming software—the components needed to extract an application's data from XML are far smaller and simpler to use than the corresponding components are for SGML.
The additional specifications, which the trade press so enjoy discussing every time a news release comes out, are generally built by defining new languages using the base XML and Namespaces recommendations. These are often documented by schema definitions (the forms that these take are described in Chapter 2) as well as committee-driven documents that attempt to explain how the language should be used. Since every industry has at least one consortium that deals in part with data interchange between different components of the industry (think of doctors, pharmacies, and hospitals in the health care field), many standards take this form. Many of the standards for XML are derived from earlier efforts using older SGML industry-specific languages, and many are new.
Locating information about the languages that have been defined for your industry may be easy or it may be difficult. There are many resources you can use to locate relevant specifications: http://www.schema.net/
This web site contains information on a range of standards based on XML, including general business-oriented specifications, industry-specific standards, interoperable languages for academic research, and general Internet-related specifications. http://www.biztalk.com/
Information about the Microsoft-sponsored "BizTalk" range of business interoperability specifications can be found at this web site. http://www.ebxml.org/
The "e-business XML" initiative, or ebXML, grows out of the EDI community, and generally competes with BizTalk. http://www.w3.org/
For general Internet-related specifications, the World Wide Web Consortium is perhaps the best place to look; the working groups there have a broad constituency and the results of their efforts have a high level of uptake wherever they apply. http://www.google.com/
IT-SC book 19 If all else fails, try searching here for "XML" and various keywords related to your industry (especially the names of major industry consortia).
1.3 The Power of Python and XML
Now that we've introduced you to the world of XML, we'll look at what Python brings to the table. We'll review the Python features that apply to XML, and then we'll give some specific examples of Python with XML. As a very high-level language, Python includes many powerful data structures as part of the core language and libraries. The more recent versions of Python, from 2.0 onward, include excellent support for Unicode and an impressive range of encodings, as well as an excellent (and fast!) XML parser that provides character data from XML as Unicode strings. Python's standard library also contains implementations of the industry-standard DOM and SAX interfaces for working with XML data, and additional support for alternate parsers and interfaces is available.
Of course, this much could be said of other modern high-level languages as well. Java certainly includes an impressive library of highly usable data structures, and Perl offers equivalent data structures also. What makes Python preferable to those languages and their libraries? There are several features, of which we briefly discuss the most important:
Python source code is easy to read and maintain.
The interactive interpreter makes it simple to try out code fragments.
Python is incredibly portable, but does not restrict access to platform-specific capabilities.
The object-oriented features are powerful without being obscure.
There are many languages capable of doing what can be done with Python, but it is rare to find all of the "peripheral" qualities of Python in any single language. These qualities do not so much make Python more capable, but they make it much easier to apply, reducing programming hours. This allows more time to be spent finding better ways to solve real problems or just allows the programmer to move on to the next problem. Here we discuss these features in more detail.
Easy to read and maintain
As a programming language, Python exhibits a remarkable clarity of expression. Though some programmers accustomed to other languages view Python's use of significant whitespace with surprise, everyone seems to think it makes Python source code significantly more readable than languages that require more special characters to be introduced to mark structure in the source. Python's structures are not simpler than those of other languages, but the different syntax makes source code "feel" much cleaner in Python.
The use of whitespace also helps avoid having minor stylistic differences, such as the placement of structural braces, so there's a greater degree of visual consistency across code by different programmers. While this may seem like a minor thing to many programmers, the effect is that maintaining code written by another programmer becomes much easier simply because its easier to concentrate on the actual structure and algorithms of the code. For
IT-SC book 20 the individual programmer, this is a nice side benefit, but for a business, this results in lower expenses for code maintenance.
Exploratory programming in an interactive interpreter
Many modern high-level programming languages offer interpreters, but few have proved as successful at doing so as Python. Others, such as Java, do not generally offer interpreters at all. If we consider Perl, a language that is arguably very capable when used from a command line, we see that it is not equipped with a rich interpreter. If we start the Perl interpreter without naming a script, it simply waits for us to type a complete script at the console, and then interprets the script when we're done. It does allow us to enter a few commands on the command line directly, but there's no ability to run one statement at a time and inspect the results as we go in order to determine if each bit of code is doing exactly what we expect. With Python, the interactive interpreter provides a rich environment for executing individual statements and testing the results.
Portability without restrictions
The Python interpreter is one of the most portable language interpreters available. It is known to run on platforms ranging from PDAs and other embedded systems to some of the most powerful multiprocessor platforms ever built. It can run on more operating systems than perhaps any other interpreter. Moreover, carefully written application code can share much of this portability. Python provides a great array of abstractions that do just enough to hide platform differences while allowing the programmer to use the services of specific platforms when necessary.
When an application requires access to facilities or libraries that Python does not provide, Python also makes it easy to add extensions that take advantage of these additional facilities. Additional modules can be created (usually in C or C++, but other languages can be used as well) that allow Python code to call on external facilities efficiently.
Powerful but accessible object-orientation
At one time, it was common to hear about how object-oriented programming (OOP) would solve most of the technical problems programmers had to deal with in their code. Of course, programmers knew better, pushed back, and turned the concepts into useful tools that could be applied when appropriate (though how and when it should be applied may always be the subject of debate). Unfortunately, many languages that have strong support for OOP are either very tedious to work with (such as C++ or, to a lesser extent, Java), or they have not been as widely accepted for general use (such as Eiffel).
Python is different. The language supports object orientation without much of the syntactic overhead found in many widely used object-oriented languages, making it very easy to define new object types. Unlike many other languages, Python is highly polymorphic; interfaces are defined in much less stringent ways than in languages such as C++ and Java. This makes it easy to create useful objects without having to write code that exists only to conform to an interface, but that will not actually be used in a particular application. When combined with the excellent advantage taken by Python's standard library of
IT-SC book 21 a variety of common interfaces, the value of creating reusable objects is easily recognized, all while the ease of implementing useful interfaces is maintained.
1.3.1 Python Tools for XML
Three major packages provide Python tools for working with XML. These are, from the most commonly used to the largest:
The Python standard library
PyXML, produced by the Python XML Special Interest Group
4Suite, provided by Fourthought, Inc.
The Python standard library provides a minimal but useful set of interfaces to work with XML, including an interface to the popular Expat XML parser, an implementation of the lightweight Simple API for XML (SAX), and a basic implementation of the core Document Object Model (DOM). The DOM implementation supports Level 1 and much of Level 2 of the DOM specification from the W3C, but does not implement most of the optional features. The material in the standard library was drawn from material originally in the PyXML package, and additional material was contributed by leading Python XML developers.
PyXML is a more feature-laden package; it extends the standard library with additional XML parsers, has a much more substantial DOM implementation (including more optional features), has adapters to allow more parsers to support the SAX interface, XPath expression parsing and evaluation, XSLT transformations, and a variety of other helper modules. The package is maintained as a community effort by many of the most active Python/XML programmers.
4Suite is not a superset of the other packages, but is intended to be used in addition to PyXML. It offers additional DOM implementations tailored for different applications, support for the XLink and XPointer specifications, and tools for working with Resource Description Framework (RDF) data.
These are the packages used throughout the book; see Appendix A for more information on obtaining and installing them. Still more are available; see Appendix F for brief descriptions of several of these and references to more information online.
1.3.2 The SAX and DOM APIs
The two most basic and broadly used APIs to XML data are the SAX and DOM interfaces. These interfaces differ substantially; learning to determine which of these is appropriate for your application is an important step to learn.
SAX defines a relatively low-level interface that is easy for XML parsers to support, but requires the application programmer to manage more details of using the information in the XML documents and performing operations on it. It offers the advantage of low overhead: no large data structures are constructed unless the application itself actually needs them. This allows many forms of processing to proceed much more quickly than could occur if more overhead were required, and much larger documents can be processed efficiently. It achieves this by being an event-oriented interface; using SAX is more like processing user-input events in a graphical
IT-SC book 22 user interface than manipulating a pre-constructed data structure. So how do you get "events" from an XML parser, and what kind of events might there be?
SAX defines a number of handler interfaces that your application can implement to receive events. The methods of these objects are called when the appropriate events are encountered in the XML document being parsed; each method can be thought of as the actual event, which fits well with object-oriented approaches to parsing. Events are categorized as content, document type, lexical, and error events; each category of events is handled using a distinct interface. The application can specify exactly which categories of events it is interested in receiving by providing the parser with the appropriate handlers and omitting those it does not need. Python's XML support provides base classes that allow you to implement only the methods you're interested in, just inheriting do- nothing methods for events you don't need.
The most commonly used events are the content-related events, of which the most important are startElement, characters, and endElement. We look at SAX in depth in Chapter 3, but now let's take a quick look at how we might use SAX to extract some useful information from a document. We'll use a simple document; it's easy to see how this would extend to something more complex. The document is shown here:
If we want to create a dictionary that maps the ISBN numbers given in the isbn attribute of the book elements to the titles of the books (the content of the title elements), we would create a content handler (as shown in Example 1-1) that looks at the three events listed previously.
Example 1-1. bookhandler.py import xml.sax.handler
class BookHandler(xml.sax.handler.ContentHandler):
IT-SC book 23 def __init__(self):
self.inTitle = 0
self.mapping = {}
def startElement(self, name, attributes):
if name == "book":
self.buffer = ""
self.isbn = attributes["isbn"]
elif name == "title":
self.inTitle = 1
def characters(self, data):
if self.inTitle:
self.buffer += data
def endElement(self, name):
if name == "title":
self.inTitle = 0
self.mapping[self.isbn] = self.buffer
Extracting the information we're looking for is now trivial. If the code above is in bookhandler.py and our sample document is in books.xml, we could do this in an interactive session:
>>> import xml.sax
>>> import bookhandler
>>> import pprint
>>>
>>> parser = xml.sax.make_parser( )
>>> handler = bookhandler.BookHandler( )
>>> parser.setContentHandler(handler)
IT-SC book 24 >>> parser.parse("books.xml")
>>> pprint.pprint(handler.mapping)
{u'1-56592-051-1': u'Making TeX Work',
u'1-56592-724-9': u'The Cathedral & the Bazaar'}
For reference material on the handler object methods, refer to Appendix C.
The DOM is quite the opposite of SAX. SAX offers a very small window of view that passes over the input document, relying on the application to infer the whole; the DOM gives the whole document to the application, which must then extract the finer details for itself. Instead of reporting individual events to the application as the parser handles the corresponding syntax in the document, the application creates an object that represents the entire document as a hierarchical structure. Although there is no requirement that the document be completely parsed and stored in memory when the object is provided to the application, most implementations work that way for simplicity. Some implementations avoid this; it is certainly possible to create a DOM implementation that parses the document lazily or uses some kind of persistent storage to keep the parsed document instead of an in-memory structure.
The DOM provides objects called nodes that represent parts of a document to the application. There are several types of nodes, each used for a different kind of construct. It is important to understand that the nodes of the DOM do not directly correspond to SAX events, although many are similar. The easiest way to see the difference is to look at how elements and their content are represented in both APIs. In SAX, an element is represented by start and end events, and its content is represented by all the events that come between the start and the end. The DOM provides a single object that represents the element, and it provides methods that allow the application to get the child nodes that represent the content of the element. Different node types are provided for elements, text, and just about everything else that can exist in an XML document.
We go into more detail and see some extended examples using the DOM in Chapter 4, and a detailed reference to the DOM API is given in Appendix D. For a quick taste of the DOM, let's write a snippet of code that does the same thing we do with SAX in Example 1-1, but using the basic DOM implementation from the Python standard library, as shown in Example 1-2.
Example 1-2. dombook.py import pprint
import xml.dom.minidom from xml.dom.minidom import Node
doc = xml.dom.minidom.parse("books.xml")
mapping = {}
IT-SC book 25
for node in doc.getElementsByTagName("book"):
isbn = node.getAttribute("isbn")
L = node.getElementsByTagName("title")
for node2 in L:
title = ""
for node3 in node2.childNodes:
if node3.nodeType == Node.TEXT_NODE:
title += node3.data
mapping[isbn] = title
# mapping now has the same value as in the SAX example: pprint.pprint(mapping)
It should be clear that we're dealing with something very different here! While there's about the same amount of code in the DOM example, it can be very difficult to develop reusable components, while experience with SAX often points the way to reusable components with only a small bit of refactoring. It is possible to reuse DOM code, but the mindset required is very different. What the DOM provides to compensate is that a document can be manipulated at arbitrary locations with full knowledge of the complete document, and the document contents can be extracted in different ways by different parts of an application without having to parse the document more than once. For some applications, this proves to be a highly motivating reason to use the DOM instead of SAX.
1.3.3 More Ways to Extract Information
SAX and the DOM give us some powerful tools for working with XML, but they clearly require a lot of code and attention to detail to use effectively in a large application. In both cases, working with complex data requires a great deal of work just to extract the interesting bits from the XML documents that contain the data. Now, what sorts of tools would we normally turn to when dealing with complex data sets? Two that come to mind are higher-level abstractions (such as APIs that do more work, and specialized task-oriented languages), and preprocessing techniques (transforming data from one form to another more suitable to the task at hand). Fortunately, both of these are available to us when working with XML from Python.
When an XML user wants to specify a portion of a document based on possibly complex criteria, she uses a language which lets her write the specification concisely; that language is called the XML Path Language, or XPath. Support for XPath is available in the 4Suite package, and has recently been added to the PyXML package as well. Using XPath, a query can be written that selects nodes from a DOM tree based on the element names, attribute values, textual content, and
IT-SC book 26 relationships between the nodes. We cover XPath in some detail, including how to use it with a DOM tree in Python, in Chapter 5.
Other times, what we'd really like is a new document that either contains less information or arranges it very differently. For this, we need a way to specify a transformation of a document that generates another document. This is provided by XML Stylesheet Language Transformations (XSLT). Originally developed as part of a new specification for stylesheets, XSLT is an XML- based language that is used to define transformations from XML to other formats. XSLT is most commonly used with XML or HTML as the output format. Chapter 6 describes this language and shows how to use it in Python.
1.4 What Can We Do with It?
Now that we've looked at how we can use XML with Python, we need to look at how we can apply our knowledge of XML and Python to real applications. In the Internet age, this means widely distributed systems operating across the Internet.
There's a lot to working with the Internet beyond XML and the CGI programming done in many of the examples in the book. In case you're not already familiar with this topic, we include an introduction to the facilities in the Python standard library that help create clients and servers for the Internet in Chapter 8. We review how to retrieve data from remote servers, and how to submit form-based requests programmatically and read the result. We then learn to build custom web servers that respond to HTTP requests, allowing us to build servers that do exactly what we need them to.
With these skills under our hat, we proceed to look at the emerging world of "web services." Chapter 9 describes what we mean by web services and introduces the specifications coming out in that area. We look at two packages that allow us to use SOAP to call on web services and demonstrate how to create one in Python.
In Chapter 10, we pull together much of what we've learned with an extended example that demonstrates how it all works together. Using XML as a communications medium, we are able to build an application that uses a variety of technologies and operates in diverse environments.
IT-SC book 27 Chapter 2. XML Fundamentals
XML is not new! XML, the Extensible Markup Language, began development in 1996 and became an official World Wide Web Consortium (W3C) standard in 1998. XML is derived from the Standard Generalized Markup Language (SGML), which has been around for a great while. SGML has long been used as a means of document management, and is the parent of HTML. XML, on the other hand, is an outgrowth of these earlier markup languages intended for information sharing on the Internet. While HTML is effective for communicating how a page should look inside a web browser, XML speaks more to how information should be structured or used between or among applications (including web browsers) running on the Internet.
2.1 XML Structure in a Nutshell
The basic structure of an XML document is simple. Most can be reduced to a few simple components. Consider the following:
In this example, the first line, starting with the and ending with is an XML element. An element must have an opening and closing tag, or the opening tag must end with the characters /> if it is to be empty. The account element shown here is an example of an empty element that ends with a />. The item element opens, contains two other elements, and then closes. The sku="33-993933" expression is an attribute named sku with its value 33-993933 in quotes. An element can have as many attributes as needed. Both the name and description elements are followed by character data or text. Finally, the elements are closed and the document terminates.
In the remainder of this chapter, we walk through the relevant parts of the XML specification, highlighting the most important items for you to be aware of as you embark on coding with Python and XML.
2.2 Document Types and Schemas
IT-SC book 28 When we talk about document types, we are speaking of something very similar to the notion of types in a programming language. Programming language types are used to describe structures that can be composed in particular ways, and document types do the same thing. The primitive components and the types of composition that are allowed differ, but they are conceptually aligned. A document type is commonly referred to as a schema. The difference between a document type and a database schema can be shallow in many applications, though the similarity is not always relevant. We often use schema to refer to a document type when it is not important how it was defined, because the phrase "document type" has historical associations with a particular schema language.
Schemas are valuable for several reasons, but two dominate: they require critical thinking about the applications and data to design, and they can be used to help specify how documents should constructed and interpreted when exchanged across organizational boundaries. The latter can be especially critical in applications such as supply-chain integration, where the automated exchange of dynamically generated documents can incur contractual obligations—it becomes very important that everyone agree what the documents mean, because misinterpretation can be very costly!
Document types are built on top of data types as well as on top of structuring rules, in which data types are very analogous to the primitive types provided by most programming languages. Different schema languages use different sets of data types, some being extensible and others allowing the use of arbitrary typing systems rather than providing their own. Some schema languages allow data types to be specified for any document content, and others limit the ability to apply data types to specific constructs.
All schema languages let the allowed ordering and nesting of elements be defined, and let attributes be associated with element types. Everything else is open to variation, so it helps to be aware of the general differences and select a schema language based on the requirements of the application, the availability of tools, and interoperability requirements.
2.2.1 Document Type Definitions
The XML 1.0 recommendation specifies one way to define a document type known as a Document Type Definition, or DTD. The language used to specify a DTD is really just part of XML itself, but is also informally known as the DTD language. This is a subset of XML that has a slightly different set of syntactic rules and does not allow arbitrary content to mix with the markup.
The DTD language for XML is derived from the DTD language for SGML, but drops many of the less commonly used constructs in favor of simplicity. The newfound simplicity pertains both for the language itself and for processing tools. The specific features that were omitted are only of interest if you already know the SGML version of the language, and so are not discussed in this book. Please refer to the XML recommendation and books focused on document type development to learn more about the differences.
We discuss the specific construction and interpretation of DTDs later in this chapter, but it is interesting to note that while the DTD language allows fairly flexible composition of elements, it defines very few data types that can be used to specify the types of attribute content, and provides almost no way to extend the set of data types. In spite of the limitations of DTDs, they are still an important type of schema due to their early specification as part of the XML 1.0 recommendation,
IT-SC book 29 their similarity to SGML DTDs, the widespread availability of tools, and the relative ease of learning how to create and use them.
2.2.2 Alternate Schema Languages
The XML sublanguage used to specify document types is largely inherited from the SGML roots of XML, and is perhaps the least appreciated aspect of the specification. The use of this language does represent a trade-off, no matter how useful it may be to particular projects. While there is no doubt that it is better than having only well-formed XML defined by the XML specification, there is a broadly perceived need for something better. As with all standards, however, one size does not fit all, so a number of alternate languages have been developed for specifying document types. Together, these are known as schema languages.
The application of each language varies, as does the level of complexity and availability of tool support. In this section, we examine some of the more popular languages and describe the intended uses for each of these, as well as what form of support is available for Python programmers. Two common aspects of the schema languages described here involve the fact that they all use XML to provide their own syntax, and they all are namespace-aware: the schema they can specify can contain elements and attributes from multiple namespaces. Both are significantly different from the DTD language, and both can easily be argued to be significant improvements.
2.2.2.1 XML Schema
The World Wide Web Consortium has been active in efforts to develop and standardize a schema language that was intended to work for everyone, and XML Schema is the result. As with all committee-driven designs, there is widespread dissatisfaction with XML Schema, not because it is not powerful enough, but because it is considered by many practitioners to be too complex. It defines ways to describe the allowed structures for a document type, as well as describe data types that can be used to describe both element and attribute content much more precisely and flexibly than what the DTD language supports.
XML Schema does offer the advantage that it provides ways to define both document types and data types, and includes a selection of basic data types to build on. These types range from numbers to strings that must match some regular expression, to more complex types such as dates or times. XML Schema data types are very rich compared to the data types supported by the DTD language. Schemas may be defined that constrain values of attributes or element content to be of these types, making it possible to describe larger document types much more precisely than the DTD language allows. This makes it possible to build tools that can validate a document against a schema, allowing application code to deal with far less specialized error-checking code. XML Schema data types are used briefly in Chapter 9, but are not discussed in detail.
There is an XML Schema validator for Python; see Appendix F for more information.
2.2.2.2 TREX
Tree Regular Expressions for XML (TREX) is a schema language designed by the notable James Clark, who has been active in developing usable XML standards for as long as XML has been around, and is known for his significant contributions to the SGML community before XML. TREX does not define fine-grain data types the way XML Schema does. It is intended to be used
IT-SC book 30 in conjunction with data types defined using external specifications, which can include XML Schema-defined data types.
The PyXML package includes a TREX validator in the xml.schema.trex module; this was added in PyXML Version 0.7.0.
2.2.2.3 RELAX-NG
RELAX-NG is a language derived from two well-received schema languages, TREX and RELAX; the specification is still under active development at the time of this writing. This specification is the combined effort of James Clark and Makoto Murata, the authors of TREX and RELAX, and is sponsored by the Organization for the Advancement of Structured Information Standards (OASIS). RELAX-NG takes the same approach to data types as TREX. Complete information on RELAX-NG is available at http://www.oasis-open.org/committees/relax- ng/. An alternate, non-XML syntax has also been proposed.
2.2.2.4 Schematron
The Schematron Assertion Language defined by Rick Jelliffe is a bit different from the other schema languages. Instead of defining what elements are allowed, their content models, and their attributes, Schematron makes assertions about the relationships among elements and attributes. Extensive documentation is available online at http://schematron.sourceforge.net/, and a Python validator is available from Fourthought, Inc. (http://www.fourthought.com/).
2.3 Types of Conformance
As with any specification, the primary reason for the XML specification's existence is to hold documents against it and make sure they conform to the specification. If so, then the rules within the specification can be used in reading, transforming, or applying the document. However, we must remember that XML defines two things: syntax for document instances, and a way to define new language using XML. It also tells us that we can use the former without the latter, so it must define what it means to conform to the specification in both cases.
If a document uses the XML syntax but does not depend on a specific markup language defined using the means provided by the XML recommendation, it needs to be well-formed in order to conform with XML. This is a form of conformance introduced by XML rather than inherited from SGML. On the other hand, a document that declares that it uses a specific markup language defined by a DTD is said to be valid if it is both well-formed and the elements and character data are arranged in a way that complies with the rules given by the specified Document Type Definition.
The XML specification defines a collection of text to be an XML document if it is well-formed according to the rules of the specification. The term well-formed is widely used in XML, and it refers to a document that is syntactically acceptable. For example:
IT-SC book 31
The preceding document is well-formed. That is, beyond the XML declaration (described in more detail in Section 2.5.6, later in this chapter) pointing out that the document uses Version 1.0 of XML, both the book and title elements are opened and closed so that elements nest within each other in a strictly hierarchical way. You can't open a book and close a magazine.
Being well-formed is required but not sufficient to describe the concept of validity, which deals with the conformance of a document to a Document Type Definition. It's one thing to have the structure arranged such that it is syntactically acceptable, but quite another to ensure that the information contained within the document is organized in the appropriate fashion and contains all of the necessary elements to be of use in an application or transaction.
The XML specification describes all XML processors as belonging to two classes: validating and nonvalidating. Regardless of validation, both types of processors must report violations of the specification's well-formedness constraints; otherwise, an XML document may be impossible to parse. A validating processor must be able to report violations of the DTD to the application. This requires that a validating processor read the entire DTD, and resolve and parse any external entities (described in the next section) referenced within the DTD itself and in the document instance. In contrast, nonvalidating processors need check only the document and internal DTD subset for well-formedness. Checking that a document is well-formed does not require accessing any external entities.
Since the arrival of alternate schema languages, a third form of conformance has been described informally. A document is said to be schema valid with respect to a particular schema, regardless of the language in which the schema is expressed, if the document is well-formed and the structure of the document conforms to the specific schema using the rules defined for that specific schema language. This is a generalization of the concept of validity given by the XML recommendation; all valid documents are also schema valid for the schema defined by their DTD (though they may be invalid for other schema).
2.4 Physical Structures
XML text is stored in entities. Entities are identified in various ways, but most commonly by filename or URI. There is no constraint on this, however, and many systems do use alternate means for entity storage — for example, many live happily in large databases. Many XML documents involve more than one entity; perhaps the most common arrangement is that the document is in one entity and its type definition is in another. As documents get larger, increasing numbers of entities are often involved with each document. This may be more common with document-centric applications than with data-communication applications of XML.
Entities are typically given names in one or more global namespaces. XML requires that entities be given system identifiers, which are always URIs. The term has roots in the SGML community, where system identifiers were used to refer to storage locations using whatever syntax the tools in use happened to understand. An additional global namespace is shared with the SGML world; the identifiers in that space are called formal public identifiers (FPIs). Use of this namespace is very limited in the XML world, as it is not always easily mapped to URLs that can be used to retrieve arbitrary resources, although there are ways to do it. They do see some use, and extensible support for FPIs is available in the PyXML toolkit.
Entities are used for several things in XML:
IT-SC book 32 Document entities
Regardless of the application, all documents start somewhere. With XML, they are also guaranteed to end in the same entity. The entity containing the start of the document is called the document entity. The document entity is interesting because it is the only entity that may be completely anonymous. An application can provide the content of the entity directly to the XML parser, allowing it to operate without extracting the text from a disk file or another local or remote data source.
External entities
Other physical storage units that contribute to a document are external entities. These entities may contain all or part of the type specification for the document, or they may contain document content. While external entities are defined by their system and formal public identifiers, most are given local names for easy reference.
External DTD subset
If a document contains a document type declaration that specifies an external document type subset, that subset is given in an entity. This entity is special in that it is not given a local name, but otherwise is simply an external entity.
Linked resources
Some documents refer to other documents without making them a part of themselves. Whether or not these external resources are really entities is not always clear if they are not referenced via a name defined in an entity declaration. One typical example of this is the resources identified by URI in the href attribute of HTML's a element—it is not referenced by a named entity, and it is not always known just what the linked resource will contain when the reference is created. The fact that the external resource is identified as the target of a link is important to the linking document.
2.5 Constructing XML Documents
Documents are the heart of XML. Any amount of usable XML is presented as a document, often stored in a file. One of the very first things you must understand in order to use XML is how to create a well-formed document. In this section, we examine the syntactic components of a document, starting with the individual characters and looking at how they are viewed when building larger syntactic constructs. Then we look at the constructs defined for all documents by the XML recommendation.
2.5.1 Characters in XML Documents
The XML Specification defines a character as "an atomic unit of text as specified by ISO/IEC 10646." (Remember, ISO/IEC 10646 is more commonly referred to as Unicode.) Of course, this explanation is exactly what you should say at a party if someone asks. One of the goals of both standardization and XML is to make documents easily understandable by platforms around the globe. As such, simple things like ASCII characters can become quite complex.
Regardless, the specification states that legal characters are "tab, carriage return, line feed," as well as belonging to the aforementioned Unicode specification. If you were to write an XML
IT-SC book 33 parser, the topic of characters and standardization would be of incredible importance to you. For the rest of us, it's usually enough to choose an XML parser that gets it right.
You can declare the character encoding used in an XML document using the optional XML declaration:
For an external entity that is not a document itself, a variation of the XML declaration, called an encoding declaration, is used:
More information on the XML declaration is provided in "The Document Prolog" later in this chapter. For now, let's look at some of the most widely used character sets and encodings. (A character set that can be mapped into Unicode can be considered an encoding of Unicode, even if it does not directly support everything defined in Unicode.)
2.5.1.1 The ASCII character set
The American Standard Code for Information Interchange (ASCII) is a 7-bit text format (meaning that it takes a sequence of seven 1's and 0's to form a character). ASCII is understood by virtually ever computer in use. Unicode extends ASCII, so the first 128 characters of Unicode coincide with the first 128 characters of ASCII.
2.5.1.2 The ISO-8859-1 character set
The character set ISO-8859-1 is also known as Latin-1. The ISO-8859-1 set is very widely used as it contains support for most (but not all) Western European languages. The first 256 characters of Unicode are identical to ISO-8859-1 for compatibility reasons. The first 128 characters of ISO- 8859-1 are identical to ASCII. The second 128 are a combination of control characters, special characters, and accented letters. ISO-8859-1 was inspired by DEC Multinational Character Set, but there are a few differences. There are also various ISO-8859-X sets with support for additional languages and characters.
2.5.1.3 UTF-8 Encoding
Universal Transformation Format, 8-bit (UTF-8), is documented in IETF RFC 2279 by F. Yergeau. UTF-8 is the most popular complete encoding of Unicode.
UTF-8 extends ASCII to some degree. The first 128 positions of UTF-8 are transparently encoded to their ASCII counterparts. Since Unicode can supposedly support over 2 billion characters (way beyond 128), getting it to fit in a stream of discrete 8-bit bytes requires some encoding. UTF-8 solves this problem by representing each Unicode character with a unique sequence of bytes. In a UTF-8 stream, ASCII characters occupy only one byte in the stream, whereas all other characters are represented by two or more bytes the stream. Your XML declaration using UTF-8 appears as follows:
The most detailed information for dealing with UTF-8 encoding comes from the RFC.
IT-SC book 34 2.5.2 Text, Character Data, and Markup
The specification states "text consists of intermingled character data and markup." The main point here is that every character within an XML document is either character data (the actual information content we're most interested in, such as an address or item quantity), or it is markup (containing all of the special characters needed to create start tags, end tags, entities, comments, CDATA delimiters, DTDs, processing instructions, and declarations). All the characters together constitute text.
Character data in the content of elements is "any string of characters that does not contain the start-delimiter of any markup." Clearly, it is important to know the difference between the two, since it is markup that allows our programs to interpret the character data correctly.
All markup begins with one of two characters: the less-than sign (<) and the ampersand (&). All markup that begins with the less-than sign ends with the greater-than sign (>), and markup that begins with an ampersand ends with a semi-colon (;). These are the only special characters you need to be aware of most of the time. In some situations, the single-quote ( ' ) and double-quote (") characters need special attention. This does not mean that your documents and data cannot include these characters, only that they require some special encoding in the XML text. Any Unicode character can be part of the character data.
One result is you're unable to use literal special characters such as ampersands (&), or angle brackets (<, >) within your text. For example, the following would confound an XML processor:
The text of the question element contains characters not allowed by the specification. The < is expected to start a new markup component, so the following space is interpreted as a syntax error. The less-than sign is used to start a variety of markup constructs, the most common of which are the element start and end tags. The ampersand is used to mark entity references.
In order to use these special characters within your XML document, you'll need to encode them using entity or character references. To turn the example into proper XML, we need to use this:
Entity references are discussed later in this chapter, although many of you who have worked with HTML will find them familiar as they include ' ('), " ("), < (<), and > (>). XML allows you to define your own entities as well, and they can contain more than a single character, but those four are defined by the XML specification and do not need to be defined specially for your documents. Character references are slightly different in that they specify individual Unicode characters without attempting to use mnemonic identifiers for them. A character reference you might have seen used in HTML would be something like ® (®, the registered trademark symbol). In XML, the numeric portion of the reference may be given using hexadecimal digits as well if the letter x is inserted between the sharp sign and the first digit. The reference ® also refers to the registered trademark symbol. Character references cannot be defined by authors, and they always refer to Unicode characters by the ordinal value assigned to them in the Unicode specification.
IT-SC book 35 2.5.2.1 Names
The XML specification defines several small lexical details, but perhaps one of the most important is the name. Names are tokens composed of some combination of legal characters including letters, digits, underscores, hyphens, or colons; the first character of a name cannot be a digit. Name tokens are used for naming anything that needs a name in XML, including element types, attributes, and entities. Some names cannot be used in day-to-day XML markup. First, names beginning with the string xml (in any mixture of upper- and lowercase) are "reserved for standardization in [the XML specification] or future versions of this specification." Secondly, when naming your elements, you must avoid use of the colon (:), as it is the basis for XML namespaces (a method of prefixing element names with tokens to give them domain context). While the XML 1.0 specification allows colons in element and attribute names, the more recent Namespaces specification assigns a particular syntactic significance that constrains their use. In other words, if you're defining a whole class of elements related specifically to books, such as bookTitle or bookAuthor, its better to use capitalization, hyphenation, or underscores to separate the words (such as book_title, book-title, or bookTitle) as opposed to using the colon, such as book:Title. Using an expression like book:Title leads XML processors to believe that you are referring to a Title element within the namespace URI attached to the local name book. Of course, it may be that Namespaces are appropriate for your application, in which case you should take the time to read the Namespaces specification very carefully and define any that are needed.
2.5.3 Whitespace in Character Data
When working with XML-based markup languages, it can be difficult to know how to treat whitespace. For many applications, whitespace can be handled as just more normal character data, while this is not sufficient for others. The problem most often manifests itself when presentation to the user is being controlled by the application. While the XML specification does not attempt to solve the problem, it does provide a way to include a hint for processing tools and applications that the whitespace in a particular element should be preserved as given, rather than treated as malleable space.
The easiest way to visualize the problem is to consider the way program source code is most commonly presented in HTML. Most HTML authors wrap source code in a pre element:
def hello( ):print "Hello, world!"
This is certainly the easiest way to present source code in HTML. Now consider what happens if, instead of using a pre element, we use a paragraph, or p, element:
def hello( ):
IT-SC book 36 print "Hello, world!"
This creates a very different effect in most web browsers, typically causing the entire program text to be shown on a single line with only a single space separating each word, even though the example includes multiple lines and multiple adjacent spaces.
The solution looks simple, at least for HTML. Simply use a pre element when we want to preserve whitespace. This obvious solution unfortunately has an equally obvious problem—it only works for HTML, not for arbitrary XML-based markup languages. A solution is needed that also works for a non-HTML document like this:
Ode to a node,
Nested beneath its tree,
Snug as a bug in its XML rug
Dreaming of the W3C.
How is an XML tool to know that the line breaks and other presentation for a poem are significant?
The XML specification defines an attribute called xml:space that you can attach to an element to communicate to the application that whitespace should be preserved. It is the responsibility of the client application to act on this information and indeed preserve whitespace when handling or formatting the data. A typical compliant XML parser passes the whitespace from the document through to the application regardless of whether the xml:space attribute has been seen (in either the document or the schema). An application can use the attribute to determine just what manipulations it can perform on the document content.
The value of the xml:space attribute can be either default or preserve. If the value is default, the application is allowed to treat the whitespace in whatever way it normally would; the XML specification imposes no limitations on how the whitespace is affected in this case. If, however, the value is preserve, the application is expected to avoid interfering with the whitespace in the element to which the attribute applies, as well as all child elements, until it encounters a child that specifies a value for xml:space. At that point, the child's value for xml:space takes precedence for itself and it's descendents.
The xml:space attribute can be used in a couple of different ways. The first is to simply include it in the document instance, which is sufficient for well-formed XML. The first line of our poem becomes:
While this seems reasonable for small quantities of XML text, it proves unworkable for large volumes of documents that are edited by humans. Think about what HTML would be like if we
IT-SC book 37 had to always include a special attribute to get the effect of the pre element! For this reason, the xml:space attribute is most often used by including it in the document schema. In a DTD, we would write something like this:
Attribute list declarations will be discussed in more detail in Section 2.6.3 later in this chapter.
From a practical point of view, most applications that parse XML look at the names of the elements to determine what to do with the character data contained therein. For example, while parsing the text of a book formatted in XML, you may come across a code element that tells you to preserve the whitespace within that section. If you look carefully, however, often the document type specifies that xml:space has a default value of preserve for those elements.
2.5.4 End-of-Line Handling
The specification is straightforward where end-of-line handling is concerned. An XML parser must pass characters to applications with normalized line endings. That is, any combination of the hexadecimal characters 0x0D and 0x0A, or a standalone 0x0D character not followed by 0x0A, is converted to a single 0x0A character. For the less hexadecimal among us, it means that typical formatting codes such as \r\n and \r are converted to \n. And for those of you who have never used those weird backslash characters, it means that text coming from platforms that commonly use carriage-returns plus linefeed characters to terminate lines (such as Windows) is converted to use only linefeed characters.
2.5.5 Language Identification
An attribute named xml:lang is provided by the specification and can be placed inside documents to indicate the language used in the content. Again, this attribute must be declared in valid documents, much like xml:space. The values that can be used within this attribute are defined in IETF RFC 1766, or in a later version. Most language character codes have two letters, such as en for English, but dialects may be specified using an underscore character and an additional two-letter code; United States English can be specified as en_US, while the Queen's English can be specified as en_GB.
2.5.6 The Document Prolog
An XML document contains a prolog, which includes everything that precedes the single element that is the document content. The prolog consists of an optional declaration called the XML declaration, followed by an optional Document Type Declaration, followed by any number (including zero!) of comments and processing instructions. So the prolog may completely empty, but often contains the XML declaration as a matter of good form. The Document Type Declaration is required if the document is intended to conform to a DTD.
The XML declaration looks much like a processing instruction, but is slightly different because of a special purpose it serves. Since XML requires that all documents are Unicode — but does not constrain the encoding of the Unicode characters to bytes in the data stream that contains the document — there must be a way to determine that encoding. Some encodings can be recognized by the leading bytes of the data stream. A set of specific rules for determining the encoding from the leading bytes of the data stream is given as part of the XML recommendation. For many
IT-SC book 38 encodings however, that is not possible. The XML specification states that in those cases where the encoding is not known a priori (as when the encoding is returned in the headers of an HTTP response), the document must be encoded in UTF-8 or include an XML declaration that specifies the encoding. The declaration always includes the version of the XML specification with which the document conforms (only XML 1.0 has been defined at this time). A typical XML declaration would look like:
This declares that the document is encoded in the character set ISO 8859-1, more commonly known as Latin-1. It's entirely legal to omit the encoding from the declaration as well, so the minimal declaration looks like this:
I'm sure this already appears on coffee mugs.
After the XML declaration, a Document Type Declaration may appear. Note that this is different from the Document Type Definition, although the first two words and obvious abbreviations are the same. To avoid confusion, the acronym "DTD" is never used to refer to this; it is usually called the "DOCTYPE declaration." If given, this declaration specifies the name of the document element, and may specify both internal and external components of the DTD. Let's look at the simplest form of this declaration:
This tells us that the document element is of the type named book, but nothing else; this is not very useful by itself. There are actually two additional components to this declaration, each of which is optional, but one or both must be provided for the declaration to be particularly useful. Let's look at an example that contains both of these components:
]>
Here, we include a specification for an external subset of the DTD (the SYSTEM and the quoted string), and an internal subset enclosed in brackets.
If the Document Type Declaration is given, the name of the document type must match the name of the root element. If you declare your document type as , then your root element must be Tool. Furthermore, all the specific relationships in the DTD concerning nesting, character data, and attributes must be enforced against the document if it is to pass the test for validity.
If you decide to use both the internal and external subsets, the internal subset overrules the external. That is, the rules contained within the DTD inside your XML document prevails over rules for the same construct in an external DTD subset.
2.5.7 Start, End, and Empty Element Tags
IT-SC book 39 An element's name communicates its type. The attributes contained within a start tag are not recognized in any particular order. The specification sees no difference between
There are several constraints to keep in mind when working with tags. First, there is a constraint on attributes: they must be unique. No attribute name can appear twice in the same start tag. Next, if the document is to be considered valid, the attributes must have been declared, and the values must be of the types specified. Additionally, attribute values cannot be, nor can they contain, external entity references. Finally, an attribute of a start tag, or its entity replacement text, must not contain the character <. As for end tags, the specification requires only that they exactly match the start tag's name. Attributes are not allowed in end tags.
Elements can contain just about any type of character data, as long as it is not confused with surrounding XML markup itself. This has been addressed earlier in this chapter in Section 2.5.2.
Empty elements are elements without content. They may contain attributes as shown in this example:
This XML represents two well-formed name elements. Both are empty, but the first expresses two attributes as well.
2.5.7.1 Quotes around attribute values
The specification defines literals as "any quoted string not containing the quotation mark used as a delimiter for that string." Functionally, literals are used to indicate the content for an internal entity and the values of attributes. Typically, attribute and value combinations look like this:
In this example, refnum is an attribute of the account element and has a value of 23908403. Either single or double quotation marks may be used, with the restriction that whichever is used to quote the value may not be directly used in the value, though it may be included using entity references or numeric character references.
As an example of an attribute value that contains both types of quotation marks, let's use this phrase:
The cat said "The dog yelled `Help!,' then I pounced."
Encoded as an attribute, we end up with this:
IT-SC book 40 'The cat said "The dog yelled 'Help!,' then I pounced."' /> 2.5.8 Comments Comments in XML are similar to comments in HTML. The specification states that comments can reside anywhere outside of other markup. A simple XML comment looks like this: Since comments are not allowed inside other markup, you can't embed a comment inside an XML start tag: This type of expression is not allowed by the XML specification. Interestingly enough, comments can appear inside a DTD. In addition, comments are not considered part of the document's character data. A couple of other caveats are that the double-hyphen (--) cannot be used inside the text of a comment as the characters --> are used to indicate that the comment is being closed. Since one of the goals of XML is to avoid the syntactic difficulties of preceding markup languages, XML simply does not allow a double-hyphen within the body of comments. Entities and other markup are not handled within the text of a comment, so you can use the characters special to the rest of XML in your comments without worry that they'll cause syntax errors in your data. The correct version of the earlier comment element is as follows: This book is about the Python programming language and XML markup language. By placing the comment inside the element instead of in the start tag, we've made it follow the rules. 2.5.9 Processing Instructions Processing Instructions (PI) allow an XML document to pass instructions to a handling application. The XML processor does not consider Processing Instructions to be part of the document's character data. The point of PIs is to hand information to an application. For example, if you are communicating an urgent piece of news and want the receiving application to present some sort of alert to the user, you might place the following instruction within the XML, so that varying applications can act accordingly (i.e., a Palm VII could beep, an X Window application could raise an alert box, and so on): IT-SC book 41 In this example, newsAlert is commonly referred to as the target; the rest of the text does not have a special name. The distinction between the two portions of the processing instruction is entirely a matter on convention; the specification mandates only the leading , and the lack of the character pair ?> within the PI. (Note that most of the APIs used to work with PIs refer to the two parts as the target and the data.) There is no specific syntax associated with the content of processing instructions, though it is recommended practice to begin each with a target (usually the name of the tool expected to handle it). It is becoming common for applications to expect the content following the target to look much like a series of attributes with values, which are commonly referred to as pseudo-attributes. Clients of this XML document are able to handle or ignore the PI in whatever way is appropriate to them. Processing Instructions are useful because they provide an XML-oriented way of passing events between applications or adding annotations to the data that are specific to particular applications. Historically, PIs were used in the SGML community to encode instructions to formatting applications, with semantics such as "add a page break here." 2.5.10 CDATA Sections A CDATA section is used to escape special characters in character data in your document. For example: This is actually an encoding of the character data: The Without using a CDATA section, this must be encoded using general entities or character references: The <ool <utter Knife & Sharpening Set The CDATA section is a good way to escape longer stretches of text that contain many characters that would otherwise be treated as markup if included directly in the text. Note that a CDATA section starts with the markup '' are encountered. Entity and character references aren't resolved or recognized, so the text does not resolve to the trademark registration symbol, though it would in normal character data or in a CDATA attribute value. 2.6 Document Type Definitions As discussed earlier, Document Type Definitions, or DTDs, are the form of document types specified by the XML 1.0 recommendation. Though there are alternatives, DTDs remain one of the most common ways of specifying a document type. In this section, we discuss the syntax of the various declarations that can occur in the Document Type Declaration; these can all appear in both the internal and external subsets. 2.6.1 Entity Declarations IT-SC book 42 Entities are sources of data that are used to compose a larger construct. Most, called general entities, are used to construct documents, but some, known as parameter entities, are used to construct the document type itself. Both are defined using an entity declaration in the Document Type Definition. Each kind of entity is defined in a separate namespace; there can be a general entity named myEntity and a parameter entity of the same name, and the names do not clash. Entities can be declared more than once — the first definition for a name takes precedence. This allows the internal subset to override a definition provided in the external subset; when used with parameter entities, this mechanism can be used to extend DTDs. Document type extension generally works best when the DTD being extended has been carefully designed with this in mind. The DocBook DTD for technical documentation is an excellent example of this. General entities can take a variety of forms: they may be parsed entities, consisting of XML text, or unparsed, such as an image stored as a Portable Network Graphics (PNG) file. The text of a parsed entity may be included in the entity declaration, or it may reside in an external source. The body of an unparsed entity is always stored externally. Most entities used with XML are parsed entities; unparsed constructs, such as images, are typically referenced using an absolute or relative URL rather than by a named entity. Parsed general entities are used to define substitution text for a (typically) shorter name. Recall that in XML, text includes not only character data, but markup as well, so the substitution can actually insert additional structure into the document as long as all structures are complete within the substitution. At production time, a parser resolves the entity into its substitution text, and evaluates the document based on how it looks after the entities have been resolved. A simple internal entity is as easy to create as a symbol and its replacement text: In your document, any reference to &sandwich; yields the replacement text of "Crabby Patty" into the document. For example: I am hungry for a &sandwich;. This sentence renders as: I am hungry for a Crabby Patty. External entities are defined using an entity declaration that gives a URL to an external resource containing the replacement text: Any reference to &legal; within a document yields: Like internal entities, external entities replace symbols with the appropriate text. Sometimes this must be done when the text uses characters that would otherwise be considered markup (such as IT-SC book 43 the use of special characters like <, >, and & in your XML). Other times, entities are used to keep boilerplate information that is normally maintained somewhere else available to the document. Parameter entities are different in both usage and applicability. They can only be used to create the Document Type Definition, and not to directly compose the document. The syntax of an XML document does not allow parameter entities to be referred to from within the document content, but only allows their use with the internal and external DTD subsets. There are no unparsed parameter entities, though a nonvalidating parser may ignore them. Validating parsers are required to parse all referenced parameter entities. The declaration for a parameter entity looks much like the declaration for a general entity, with just a couple of additional characters added: What this declaration has that the general entity declaration doesn't is a percent sign (% ) between the keyword ENTITY and the name of the entity, with whitespace on both sides to set it off (the whitespace is required). This parameter entity would be used like this: %node-decls; Note that the reference to the parameter entity uses the percent sign instead of the ampersand to mark the beginning of the name; this is necessary since the two sets of names may overlap. The effect of entity replacement is much like the use of general entities. The replacement text effectively replaces the entity reference, and interpretation of the document type continues using the modified text. The usefulness of parameter entities is highest when working with modularized document types, which can provide carefully designed extension mechanisms using parameter entities. A large DTD, such as the industry-standard DocBook DTD for software documentation, can be customized by creating a new document type that simply defines several parameter entities and then incorporates the standard DocBook definition. Since the entity declarations in the customization layer override the definitions provided by DocBook, this mechanism can be used to either extend or restrict the specific document type in ways that are suitable for a specific project. 2.6.2 Element Type Declarations Element type declarations are used to constrain an element's content. They indicate what element types can be used as children of the element, and show how the children may be arranged. Element type declarations may look like this: IT-SC book 44 We can break up the declaration in particular systactic components, each with a specific purpose: The text It is interesting to note that there is not a place for attributes to be declared. While attributes are associated with element types, they are defined using attribute declarations, described later in this chapter, in Section 2.6.3. 2.6.2.1 Content models A content model describes what elements are allowed as children of the declared element type, in what order and combination they are allowed, and whether arbitrary character data is allowed. The content models of all elements can be broken into two categories: Element Content This describes content made up only of elements. That is, you define an address element that requires no character data, but instead requires child elements. The specification defines content particles that "consist of names, choice lists of content particles, or sequence lists of content particles." Mixed Content This content may contain character data. This is the most common arrangement in text documents: This article describes XML transmissions from outer space. outer space is not a meteor. In this example, elements and character data are mixed beneath the news element. Elements that have a mixed content model are not required to allow other elements as content. In fact, an element type with only character data in the content model may be completely empty; there is no way to specify that there must be characters in the character data. IT-SC book 45 Let's take another look at our example element declarations: These element type declarations are simple. The content model of the first, EMPTY, can be used to describe an empty br element as found in XHTML. It can contain no child elements and no character data. It can still contain noncontent constructs, such as comments or processing instructions. An element type declared as EMPTY is considered a degenerate special case of element content. Next, we have an element named generic that can contain any kind of element defined in the document type (this does not allow undefined element types!). In addition to other elements, character data is allowed as well, so a content model of ANY is mixed content. The third example is simple, but very different from the others. Instead of a simple name such as ANY or EMPTY, the model is described by something that closely resembles a regular expression. In this particular example, we have a name element that requires one or more address elements to be included. This form of content model is perhaps the most commonly used and allows for fine control. Content models can take on varying levels of complexity, but the goal is always the same: to define the content that is allowed or expected within the element. The content model is specified with parentheses, as well as with commas indicating a sequence. Vertical bar characters (|) indicate a choice. For example: This element type requires a first child element followed by a last child element, and nothing else. If you want to offer a choice between first or last, but not allow both, use a vertical bar: These expressions can be nested within each other as well: The above order element requires a child sku element, followed by a quantity element, then followed by either an account or a name element, and finally followed by a price element. Additionally, the operators +, *, and ? can be tacked onto the end of content expressions to indicate the number of times an element or sequence must occur, or whether it is repeatable or even required. Without a modifier, the element must appear exactly once in that location. They are explained in the following list: + Content must appear one or more times. * IT-SC book 46 Content may appear zero or more times. ? Content may appear zero times or one time. For example, to require an order element to have only one account, followed by at least one or more skus, contain one or more price elements, and optionally provide a shipping address (ship) once only, you could use an Element type such as the following: To mix a combination of character data or elements, you can use the or operator to specify your mixed content, as shown here: This paragraph element type allows for repeatable sequences of character data (denoted by the asterisk), list elements, or picture elements within paragraph elements. #PCDATA can only be combined with elements using the or operator in a group that has a * modifier, and it can only occur in the outermost parenthesized group of a content model. 2.6.3 Attribute Declarations As discussed earlier, attributes are used to provide name/value combinations as properties of elements. Attributes can appear only in start tags and empty element tags. An attribute-list declaration would be a part of a DTD, used to validate the XML document. An example follows: title CDATA #REQUIRED author CDATA #IMPLIED> This is an attribute-list declaration that indicates that any news element is required to have a title attribute consisting of character data, and may optionally have an author attribute, also consisting of character data. 2.6.3.1 Attribute data types The specification states that attribute types are of three kinds: string, tokenized, and enumerated. In the earlier attribute list example, you saw that a news element required a title attribute with the string type CDATA. There are several tokenized attribute types: ID A unique identifier for this element. The identifier must be a name unique in the current document instance. IT-SC book 47 IDREF Must match an ID somewhere in the XML document. IDREFS A list of one or more names, separated by spaces. Each must match an ID in the document. ENTITY Matches the name of an unparsed entity declared in the document. ENTITIES A space-separated list containing one or more entity names. NMTOKEN The most seldom used, this matches an NMTOKEN production as defined in the XML recommendation; refer to the recommendation for more information. NMTOKENS A list of one or more space-separated NMTOKEN values; this is the least used attribute type. The remaining attribute types, the enumerated types, are defined in the attribute list itself. An enumerated type is a type that takes a name from a defined list of names, in which the list is given in an attribute declaration. Each distinct set of names forms a separate type, but these types do not have names of their own. An example should help clarify this: type (sloop | frigate | dinghy) #IMPLIED> This declaration defines an attribute type that may have a value of dinghy, frigate, or sloop, but no other value. The element 2.6.3.2 Attribute values and constraints An attribute declaration allows the document type to specify a default value for an attribute if the attribute is missing. It can also indicate whether the attribute may be omitted from the document. Let's look at a more interesting example of an attribute declaration: synopsis CDATA #IMPLIED author CDATA #REQUIRED email CDATA "[email protected]" IT-SC book 48 version CDATA #FIXED "1.0" type (normal|reference|appendix) "normal"> The synopsis attribute is required to be a string (CDATA) if it is given at all, but it is not required, and does not have a default value because it is marked as #IMPLIED. (Most of the attributes in HTML are declared this way.) The #REQUIRED constraint means just what it says; the author attribute must be specified in the document. Because it is a string, it may be empty. If a string value is specified instead of #IMPLIED or #REQUIRED, as with the email attribute in our example attribute list, it becomes the default value that is used if no value is given in the document. The #FIXED constraint can only be used in conjunction with a default value, which we see for the version attribute. When this constraint is used, the document is allowed to include the attribute, but the value must match that given by the default exactly, though it may be encoded using a different mixture of characters, entity references, and character references. If the value differs, an error is reported by the parser. The type attribute is an example of an enumerated type, similar to what we looked at earlier. Default values and constraints are specified for enumerated types in the same way as for other types, with the additional constraint that if a value is specified, it must be one of the names included in the enumeration. ID attributes offer some unique behavior. Let's create an attribute for the news element we defined previously: newsID ID #REQUIRED> With this attribute list, news elements are required to have a newsID attribute. The allowed values are governed by the rules of the ID tokenized type. Specifically, the ID value is a name (as defined in this chapter in Section 2.5.2.1) and must not appear more than once in an XML document as the value of any attribute of type ID. In other words, ID values must uniquely represent an element within the document. Consider a legal example: Since the values of ID attributes are required to be unique within a document, the following is illegal: Additionally, no element may have more than one ID attribute specified. An element type may define more than one attribute of the ID type, but at most, one ID value may be specified for any element. As a result, some of the programming APIs can use the values of ID attributes to retrieve specific elements from a document. IT-SC book 49 What is most interesting about ID attributes, however, is not the attributes themselves, but the IDREF attribute type. While a particular value may only appear once in a document as an ID type, it may appear any number of times as the value of an IDREF or IDREFS attribute. In particular, attributes of those types may only take values that also appear as the value of an ID attribute somewhere in the same document. (An IDREFS attribute can take a value that is a space- separated list of ID values, each of which must exist in the document.) These values can be used to forge internal links between elements that a validating parser must check. This can be very convenient when a basic tree structure is not sufficient to model your data; the ID, IDREF, and IDREFS attributes can be used to extend the data model to include connected, directed graphs with typed arcs. 2.7 Canonical XML The term canonicalization originally was "borrowed" loosely from its more ancient context to indicate that one structure of an instance document is the same as the master, or commonly accepted, structure of the document. Canonicalization is sometimes referred to as C14N for brevity; this is similar to the more common use of I18N for internationalization. Canonical XML is an emerging W3C recommendation that allows you to see if one physical representation of a document is equivalent to another physical representation of the same document in order to determine if they are "canonically" equivalent. In this section, we explore some of the technical features of Canonical XML to gain a better understanding of its application to suit your needs. 2.7.1 The Canonical XML Data Model To begin the process of converting a document to canonical form, you, or rather your Canonical XML processor, must start with some form of XML that it can understand. Therefore, your first parameter to a canonical translator should be an XPath node set, or a serialized XML document. The second parameter is a Boolean value, which indicates whether comments should be analyzed. In the case of a node set, it must have normalized line feeds, normalized attribute values, substituted CDATA sections with their character content, and resolved character and parsed entity references. In other words, each node must be fully cooked. No stranded entities and no superfluous whitespace are allowed. All whitespace within the root element must be preserved with the exception of line- delimiter normalization. The whole approach leads you to think that the document is being worked over—flattened, stretched, and pulled like pizza dough just prior to being cooked. 2.7.2 Document Order Although Canonical XML depends on XPath, it imposes a few rules on the XPath node sets that are sent into any Canonical XML processor. An element's namespace and attribute nodes must follow the element but precede any children. Namespace nodes must exist prior to attribute nodes. IT-SC book 50 Namespace nodes for an element are sorted lexicographically by local name. Attribute nodes for an element are sorted lexicographically with the namespace URI as a primary key and the local name as a secondary key. 2.7.3 Canonical XML Structure Canonical XML does away with the XML declaration and DTD, and also normalizes whitespace outside of the root element. Abbreviated empty elements (in the style of Canonical XML is quite new, and we have yet to see significant amounts of Python software developed for Canonical XML processing. The vision of Canonical XML is blurry, but it is a method for checking two instances (regardless of DTD or Schema) and working them over like cleaned fish to see if they share the same skeletons. Version 0.7 of PyXML will include support for rendering XML in canonical form. 2.8 Going Beyond the XML Specification The standards developed at the W3C ensure interoperability between distributed systems and the applications developers around the world. As we progress in this book from XML tools and strategies in your local applications to distributed application development, several new XML terms and issues come into the forefront. 2.8.1 XML Namespaces As discussed in Section 1.2.2 in Chapter 1, namespaces provide a means to combine elements from different knowledge domains or schemas. The Namespaces specification accomplishes this by allowing element and attribute names to be qualified with a URI; every URI corresponds to a unique namespace. Namespaces are used for several purposes in practice, but the most important is to allow a document to contain elements defined by different schema (possibly originating from different organizations) without having naming conflicts. Namespaces are used by associating a named xmlns attribute with a URI. Namespaces are communicated in an XML document using the reserved colon character in an element name, prefixed with the xmlns symbol. For example: xmlns:sumc="http://www.superultramegacorp.com"> IT-SC book 51 In this document, the namespace of SuperUltraMegaCorp is defined. The prefix sumc has been associated with it in the xmlns:sumc attribute. Elements prefixed with sumc: are within this namespace. This purchaseOrder now has a context that can set it apart from a similarly structured purchase order intended for a different business domain. 2.8.2 Extracting Information Using XPath XPath is discussed at length in Chapter 5. For now it is worth a mention, lest you start to develop your own method for querying XML without understanding what standards are offered. XPath offers a standardized method of querying XML for specific information, whether it's a single element or node, or a collection of elements. The standardization is of value not when you're writing the backend part of your application, but rather when you need to expose search capabilities either programmatically or via the web. 2.8.3 Using XLink to Link XML Documents The XLink language allows for the insertion of elements into XML documents to create and describe links between different resources. XLink uses XML syntax to create structures representing links similar to hyperlinks used in HTML, as well as more complex linking structures. Link specifications are encoded in the attributes of the source document, or in supplemental documents that can describe links among other documents. The most common applications embed link information at the link source. The target of a link is described using a URI and an XPath expression; the URI specifies the target resource, and the XPath expression specifies a specific location in the linked resource. XLink is still a young specification and is not discussed further in this book. 2.8.4 Communicating with XML Protocols The XML Protocol working group is a W3C group tasked with investigating the development of XML-based messaging and communications standards. These standards are attempting to define a method of packaging information and sending it across the Internet. Some are focused on transactions, some are focused on guaranteed delivery, and others are focused on routing and enveloping mechanisms. The Protocol Activity page (http://www.w3.org/2000/03/29-XML- protocol-matrix) is an excellent online resource for comparing these different protocols when developing distributed systems. The Web Distributed Authoring and Versioning specifications from the IETF, collectively known as WebDAV, use XML to support interoperable tools for web site management and authoring. Chapter 9 covers such items as remote procedure calls and web services (including SOAP) in greater detail. Additional specifications deal with other aspects of distributed computing, especially topics such as authentication and secure communications. 2.8.5 Replacing HTML with XHTML IT-SC book 52 The Extensible Hypertext Markup Language, or XHTML (http://www.w3.org/TR/xhtml1/), is a welcome gift to those of us who have had to struggle with parsing HTML. Though there is a W3C specification for HTML, most implementations conform only partially. This is due in part to the growth of HTML from some early implementations rather than a formal specification, and also to the browser implementers' attempts to do "the right thing" even with badly broken markup. The attempts to force HTML to fit into an SGML mold after the fact probably hindered compliance further, if only because the rules for parsing it became more complex and implementers' don't like to start over. When a browser parses HTML, it concerns itself with display attributes, not organization of the information in the document. While XHTML doesn't change the focus on appearance, it is an XML-based markup language, allowing you to parse it with an XML parser. This can drastically reduce the handling time of XHTML. It also allows you to leverage XHTML into other XML applications, as well as use XML Namespaces in conjunction with XHTML that has migrated into other domains and systems. The first version of the XHTML specification, XHTML 1.0, defines a monolithic document type that corresponds closely with the HTML 4 specification. Future versions of XHTML, starting with XHTML 1.1, are moving toward a modular approach; different aspects of the language will be defined in separate components, and different applications will have the flexibility to determine which components they support. Part of the intent is to allow browsers with simpler displays, such as mobile phones, to avoid having to implement portions of XHTML that do not make sense for the application (such as tables for very small textual displays). An additional benefit is that application developers can define new modules that allow documents to be created that can be used for both presentation to people and improved computer-to- computer communications. 2.8.6 Transforming XML with XSLT The XML Stylesheet Language, or XSL, consists of two component specifications: XSL Transformations (XSLT) and XSL Formatting Objects (XSL-FO). The transformation language is used to translate XML documents from their original form to some other form, which may be XML, HTML, or anything else (including plain text). XSLT is covered in more detail in Chapter 6. The XSL-FO specification describes specific presentational styling and is used to describe a formatted document that could be printed to a typesetting device or displayed on a screen. It is not as widely implemented as XSLT and is not covered further in this book. IT-SC book 53 Chapter 3. The Simple API for XML The Simple API for XML, otherwise known as SAX, is a popular interface for working with XML data. Let's start by looking at the background and history of SAX, after which we'll describe the major components of the interface. Once the overview is complete, we can look at several examples to help you see how to use it in your own applications. 3.1 The Birth of SAX Before SAX, almost every XML parser offered its own interface, so applications were built to use specific parsers. The interfaces were low-level and generally similar in structure; the differences were mostly in the details. When new parsers were made available, applications had to be modified extensively to work with the different interface in order to take advantage of the new parser, even though the fundamental structure was essentially unchanged. As is so often the case, the solution lay in introducing another layer of indirection. A group of XML developers using Java, led by David Megginson on the XML-DEV mailing list, defined a set of Java interfaces that allowed an application to work with any parser. The only requirement was that there be a driver for the new API for each parser. The driver was a class that used the parser-specific interface to make calls back to the application using the new, general interface. The application would create handler objects that implemented methods the driver would use to call back to the application. When Megginson released the specification, he also released a set of drivers for many of the more popular Java XML parsers. The initial specification supported the XML 1.0 recommendation, but not any of the more complex layers that have been built on top of it; the initiatives to create those were largely in their infancy at the time. The group of developers called the new API the "Simple API for XML," or SAX, because it was actually simpler than most of the parser-specific interfaces it was designed to abstract away. The new API was widely received as a major step forward for application writers—it was easy to use, allowed the use of arbitrary parsers with an application, and was carefully defined before any other common APIs were available. Java programmers became extremely happy as the stress levels dropped in their professional lives. Developers in other languages adapted the specification in ways that allowed SAX to remain an identifiable API even as it was made to work with the native conventions used in those languages. Python programmers in the XML-SIG, led by Lars Marius Garshol, created an adaptation of the API and implemented drivers for several parsers. This implementation was accepted as part of the PyXML package. The W3C then released the Namespaces recommendation. This recommendation changed the very concept of what constituted a name. While there was great debate over the value of the new recommendation, most people recognized that it did solve real problems and that it was here to stay. No one wanted a return of having to chase incompatible APIs, so the SAX developers quickly dug in and worked on a version of SAX that could support Namespaces. The revised API is known as SAX2. It is interesting to note that some of the first implementations of namespaces were filters written as SAX handlers; the SAX events were used to drive the SAX2 handlers with a little bit of processing in the middle to add the Namespaces support. Information on the Java version of SAX2 and links to additional SAX resources can be found at the SAX home page at http://www.saxproject.org/. IT-SC book 54 Python developers rapidly adopted the SAX2 interface, taking the opportunity to clean up some warts of the early mapping of SAX from the Java-based specification. The SAX2 API rapidly became part of PyXML and was adopted for use in the Python standard library. When Python programmers speak of the SAX API, they are generally referring to the second version. 3.2 Understanding SAX The first job of using SAX is to design and implement a handler that works with your specific XML documents. When dealing with a large project or working with a vast catalogue of valid documents, it may make sense to implement a few comprehensive handlers to deal with multiple document types. However, for smaller projects, it may be more desirable to implement handlers for each specific document type that you encounter. As you start to build more complex applications, you will see that the things you're attempting to do with the XML as well as the XML documents themselves can drive the way you develop your document handlers. Often, the SAX methods that you implement extract data from the event stream, which you can then hand off to another application (such as a database). Or you might want to apply intelligent business logic to it. It's likely that the task will drive your development strategy. In all practical use, SAX is a callback-based API in which you implement handler objects to process XML. You pass a reference to your SAX handler objects to a SAX- capable parser (or driver; we'll use "parser" to refer to either). When parsing begins, the parser calls the methods on your handler objects and allows you to process the XML, so that you can do something useful with it in your applications and distributed systems. SAX is an excellent stream-based API. It allows for faster processing of documents, as well as handling of documents that are simply too large to load into memory. Additionally, the event-based API allows you to react to parsing events and errors in "real-time," as they occur, while parsing the document, rather than waiting for the entire document to load. This can be especially valuable when used in a graphical application that needs to remain responsive to the user. Another huge win for many applications is the lower memory consumption when compared to DOM-based code; by allowing the application control over any objects created during parsing, the application can minimize the needed storage overhead and discard objects as soon as they are no longer required. SAX is the interface to use when you need to construct some application-specific data structures from one or more documents, but you don't need to maintain the XML structure within your application. Since SAX reports low-level events to the handlers installed by the application, the programmer needs to be careful about keeping track of the application state during parsing—it lends itself toward modeling the application as a state machine. Fortunately, the programmer is not required to pay a high memory or a performance penalty, which is often associated with loading potentially large documents. This would be difficult to avoid when using the DOM interface, which usually keeps the entire document tree in memory until the tree is discarded. (We look at the DOM in detail in the next chapter.) 3.2.1 Using SAX in an Application IT-SC book 55 When an application is built using SAX, it can be helpful to think of the application as a set of components. The XML parser itself, including the SAX driver, is a black-box component that only needs a small amount of control information from the application. The handler objects are the only way for the XML parser to communicate with the application, but the logic they contain should be more concerned with interpreting the events reported by the parser than in implementing the application—these often form a separate layer that provides the application with the data model it needs. The application itself uses the derived data structures and higher- level events from the handler objects to perform the real work of the application. The relationship of these components is shown in Figure 3-1. Figure 3-1. Components of a SAX application For smaller applications, it is common for the application and the handlers to be the same objects, often with the application code in the callback methods. While this does not work well for larger applications, it is a reasonable approach for simple applications. While learning about SAX, it offers excellent pedagogical side effects as well, so our examples embed the application code directly in the handler implementations. It is not difficult to see how to create abstractions between the SAX handler objects and a larger application. SAX refers to the parser object as a reader. It reads input from some source and generates calls to the handler methods for particular events in the input. (There isn't any requirement that the source be an XML document, though it usually is.) The application registers handler objects using methods on the reader, and may set some additional properties of the parser. In our overview of the API, we start by examining the handler objects that can be provided to the parser and then take a quick look at the reader interface. 3.2.2 SAX Handler Objects SAX is composed of four primary interfaces that are called by parsers for the different events that are encountered during the parsing phase. Python has tailored these methods slightly (mostly by using Python's more powerful native data types) from its native Java to faithfully implement SAX in the Python environment. By implementing the different interfaces of the callback API, you can receive all the events generated by the parser as it encounters the different parts of the XML document. Let's take a quick look at the different handler objects that can be implemented. (Complete reference information on the methods invoked by the parser for each object is given in Appendix C.) 3.2.2.1 ContentHandler IT-SC book 56 The ContentHandler interface is the most commonly used of all SAX interfaces, and is the primary way in which your applications receive parsing events. Parsing events are geared towards the primary markup and character data present in documents. Tell your SAX-capable parser about your implementation of this interface via the setContentHandler method. The callback API is the part of SAX that users of XML are most interested in. This is the API that you implement to receive the stream of events generated by the parser. As each element comes through, it triggers the parser to call a startElement method on the handler you implemented. The startElement handler, designed for the XML in use, must know what to do with any element it encounters in the document: def startElement(self, name, attrs): if name == "webArticle": subcat = attrs["subcategory"] if subcat.find("tech") > -1: self.inArticle = 1 self.isMatch = 1 elif self.inArticle: if name == "header": self.title = attrs["title"] self.inBody = 1 if name == "body": self.inBody = 1 3.2.2.2 ErrorHandler The ErrorHandler interface allows applications to respond to errors encountered by the parser at runtime. This object must be registered with the reader object (using setErrorHandler) to be effective. All parse errors are classified into three categories based on their severity; the handler object implements a different method for each level of severity. The least severe errors are passed to the warning method, while real violations of the specifications are passed to the error method if the parser can continue to look for additional errors in the input. They are passed to fatalError if this is not possible. Each of these methods receives a single parameter, which is always an instance of the SAXException interface. This interface offers a number of methods to allow information about the error to be retrieved, including where the error occurred and in which input source. If the handler decides to terminate processing, the SAXException object can simply be raised as an exception. IT-SC book 57 If you do not supply an error handler, the default behavior is to print an error message to sys.stdout for warnings, and to raise the exception for both normal and fatal errors. If you have installed the PyXML package, a couple of convenient implementations are provided in the xml.sax.saxutils module. The ErrorPrinter class is an error handler that prints a report of the error on standard output, regardless of the severity. The ErrorRaiser simply raises the exception, so errors always terminate processing. 3.2.2.3 DTDHandler When an application needs to know about notations and unparsed entities, it can use the SAX parser's setDTDHandler method to specify a DTDHandler object to receive this information. Objects with this interface need only implement two interfaces—one to receive notation definitions, and one to receive entity definitions. Only definitions of unparsed entities (entities with specified notations) are passed to this interface. While this doesn't sound like it covers much of the information specified in a DTD, it does cover what an application is normally expected to need if using unparsed entities. Remember, the "S" in SAX stands for "Simple"—most applications do not actually need the details of the content models and other entity definitions. If you do need more information from the DTD, many mechanisms are available: The optional SAX DeclHandler handler, which may not be supported by all parsers The native interface of the Expat parser; see the documentation for the standard library module xml.parsers.expat The xml.parsers.xmlproc.dtdparser module from PyXML 3.2.2.4 EntityResolver This handler, if implemented, must also be registered with the parser prior to parsing, using the parser's setEntityResolver method. When the parser encounters external entities, it calls the resolveEntity method in your implementation. Application developers can use this method to point the parser at an alternative location to resolve entities, such as a cache. If it returns None or a system identifier, the parser tries to load the entity using the basic facilities for HTTP and FTP provided by the Python standard library. 3.2.2.5 Other handler objects There are actually two more handler objects defined for use with SAX, but these are considered optional and do no have methods on the parser to set them as conveniently. Most applications will not need these, but being aware of them helps when they are needed. DeclHandler An object with methods that are called when the parser encounters definitions of the structural model of the document. The methods are called for element and attribute declarations, and for declarations of both internal and external entities. IT-SC book 58 LexicalHandler The methods of this object are called for events that applications are not supposed to care about, but that can be useful when performing a transform that should not affect the document any more than necessary. The events reported to this handler include comments, entity boundaries, the start and end of the DTD, and CDATA section boundaries. There are no setDeclHandler or setLexicalHandler methods on a SAX parser. These handlers are installed using the property interface of the parser, which we discuss shortly. 3.2.3 SAX Reader Objects To use the handler objects, we must register them with a SAX reader, or parser. All parsers are required to support the four most commonly needed handlers, and convenient methods are defined to set and retrieve the values of each of these. The routines setContentHandler, setDTDHandler, setEntityResolver, and setErrorHandler all have matching routines to retrieve the current handler; these methods have names that start with get instead of set. There is an additional method, setLocale, which can be used to specify the locale for errors and warnings. In addition to these configuration methods, SAX provides the concepts of features and properties. A feature is some bit of functionality that may be turned on or off, and a property is a named value associated with the parser's state. Depending on the specific feature or property and the parser implementation, each may be either read-only or modifiable, or perhaps modifiable only when a parse is not in progress. The DeclHandler and LexicalHandler discussed previously are configured by setting properties on the parser. Most applications will not need to use properties or features. 3.3 Reading an Article In this example, we look at how we can extract and use information from an XML document using SAX. The particular documents our script works with are simple news articles, but we'll see how to work with elements, attributes, and textual content. Some of the trade-offs of using SAX depend on what you're trying to accomplish, and how the XML is structured. SAX treats XML as a continuous stream, firing events to your handler as they happen. Example 3-1 shows article.xml. Example 3-1. article.xml length="3k" author="Joe Reporter" IT-SC book 59 distribution="all"/> announced that NASA has completed building a Warp Drive and has parked a ship that uses the drive in his back yard. This individual claims that although he hasn't been contacted by NASA concerning the parked space vessel, he assumes that he will be launching it later this week to mount an exhibition to the Andromeda Galaxy. Example 3-1 contains markup that is structured in a few different ways, and can be interesting to parse via SAX. A document such as article.xml requires that we understand how the document is structured prior to writing a handler to parse it. Therefore, the handler is tightly coupled to the document's structure. 3.3.1 Writing a Simple Handler You can write the ArticleHandler class to a new file, handlers.py; we'll keep adding new handlers to this file throughout the chapter. Keep it simple at first, just to see how SAX works: # - ArticleHandler (add to handlers.py file) class ArticleHandler(ContentHandler): """ A handler to deal with articles in XML """ def startElement(self, name, attrs): print "Start element:", name Now we need to create a script to instantiate the parser, assign the handler, and do the actual work. 3.3.2 Creating the Main Program No matter how complex your handler objects become, there is rarely much code involved in setting up the parser. Let's look at Example 3-2, in which we use only the ArticleHandler IT-SC book 60 class just created, and parse what we find on the standard input stream. The file art.py, shown in Example 3-2, demonstrates how to do this. Example 3-2. art.py #!/usr/bin/env python # art.py import sys from xml.sax import make_parser from handlers import ArticleHandler ch = ArticleHandler( ) saxparser = make_parser( ) saxparser.setContentHandler(ch) saxparser.parse(sys.stdin) Once created, you can run the code from the command line using file redirection to populate standard input (both Unix and Windows): $> python art.py < article.xml The output using the simple article handler appears as: Start element: webArticle Start element: header Start element: title Start element: body The output reflects the simple rule in your ArticleHandler class, which just prints out the name of each tag it encounters. To really use the XML, you have to add more functionality to the handler class in the handlers.py file. 3.3.3 Adding Intelligence IT-SC book 61 XML allows information to be parsed for different purposes. If you create a news article in XML, one application can grab it and display it as HTML, while another can index it to a search database. It's easy to imagine that a service might like to offer intelligent agents to scour Internet sources for news items, special offers, and other items of interest for you based on preferences that you set up. XML makes this process manageable, as opposed to the alternative of reliably parsing HTML for structured information, which is nearly impossible. HTML only communicates the appearance of a document and not its organizational structure. In HTML, two documents may look exactly alike in the browser, but use wildly different tags under the hood. Parsing the HTML for its information won't work, unless of course the page designer had that goal in mind when setting out to create the page. Your news agent is configured to go after technology stories, especially ones that relate to space travel. When it discovers such an article, it displays a message, the headline, and the first few words of the body text. You can add functionality to your handler class to support this. Since SAX is stream-based, it's sometimes necessary to set flags so that you can track when you've entered certain elements in and when you haven't. If you find that you're setting too many different flags, you might consider using a DOM approach as opposed to SAX. SAX is perfect when doing bulk operations on a lengthy XML stream. However, if you are trying to pull a complex data structure out of the document, you may be better off using the DOM. To keep our example simple, set a few flags as the events are propagated, and go after the desired information. In the startElement method, check to see if you're indeed inside a news article and if your article is indeed technical. If it satisfies both of these requirements, change a Boolean data member so that other methods start paying attention to the data they receive. Also set a property on the handler itself so that the main application knows the handler has found a technical article, as that was its assignment: def startElement(self, name, attrs): if name == "webArticle": subcat = attrs.get("subcategory", "") if subcat.find("tech") > -1: self.inArticle = 1 self.isMatch = 1 elif self.inArticle: if name == "header": self.title = attrs.get("title", "") if name == "body": self.inBody = 1 IT-SC book 62 The last conditional test is to see if the parser has entered the body element of a relevant article. If so, the characters method now knows to begin buffering data as the it is called: def characters(self, characters): if self.inBody: if len(self.body) < 80: self.body += characters if len(self.body) > 80: self.body = self.body[:78] + "..." self.inBody = 0 Finally, look for the close of the body tag to indicate to the characters method that it no longer needs to pay attention to character data: def endElement(self, name): if name == "body": self.inBody = 0 Beyond implementing these three methods, the class is also modified to initialize data members, and to provide an isMatch data member to indicate to the main application whether this handler has found something worth keeping. The complete class (replacing the earlier class of the same name) is shown in Example 3-3. Example 3-3. Enhanced ArticleHandler from XML.sax.handler import ContentHandler class ArticleHandler(ContentHandler): """ A handler to deal with articles in XML """ inArticle = 0 inBody = 0 isMatch = 0 title = "" body = "" IT-SC book 63 def startElement(self, name, attrs): if name == "webArticle": subcat = attrs.get("subcategory", "") if subcat.find("tech") > -1: self.inArticle = 1 self.isMatch = 1 elif self.inArticle: if name == "header": self.title = attrs.get("title", "") if name == "body": self.inBody = 1 def characters(self, characters): if self.inBody: if len(self.body) < 80: self.body += characters if len(self.body) > 80: self.body = self.body[:78] + "..." self.inBody = 0 def endElement(self, name): if name == "body": self.inBody = 0 3.3.4 Using the Additional Information Now that the handler has been modified to collect more information and determine if the article is interesting, we can add a little more code to art.py so that when an interesting article is found, it IT-SC book 64 prints a report for the user and ignores everything else. To do this, we need only append this code to the end of art.py, which was originally shown in Example 3-2: if ch.isMatch: print "News Item!" print "Title:", ch.title print "Body:", ch.body With article.xml as input, you should see the following output: $> python art.py < article.xml News Item! Title: NASA Builds Warp Drive Body: Seattle, WA - Today an anonymous individual announced that NASA has completed building a... 3.4 Searching File Information In this section, we create a file indexing script that can generate an XML document representing your entire filesystem or a specific portion of it. Indexing files with XML is a powerful way to keep track of information, or perform bulk operations on groups of particular files on a disk. You can create an XML-generating indexing routine easily in Python. The index.py program in Example 3-4 (which shows up a little later in the chapter) starts in any directory you specify and generates an element for each file or directory that exists beneath the starting point. Once we have the index of file information, we look at how to use SAX to search the information to filter the list of files for whatever criteria interests us at the time. 3.4.1 Creating the Index Generator The main part of this routine works by just checking each file in a starting directory, and then recursing into any directories it finds beneath the starting directory. Recursion allows it to index an entire filesystem if you choose. On Unix, the program performs a lot of work, as it does content checking via a popen call to the file command for each file. (While this could be made more efficient by calling find less often and requiring it to operate on more than one file at a time, that isn't the topic of this book.) One of the key methods of this class is indexDirectoryFiles: def indexDirectoryFiles(self, dir): """Index a directory structure and creates an XML output file.""" IT-SC book 65 # prepare output XML file self.__fd = open(self.outputFile, "w") self.__fd.write(' XML_ENC + '"?>\n') self.__fd.write(" # do actual indexing self.__indexDir(dir) # close out XML file self.__fd.write("\n") self.__fd.close() An XML file is created with the name given in outputFile and an XML declaration and root element are added. The indexDirectoryFiles method calls its internal _ _indexDir method—this is the real worker method. It is a recursive method that descends the file hierarchy, indexing files along the way. def __indexDir(self, dir): """Recursive function to do the actual indexing.""" # Create an indexFile for each regular file, # and call the function again for each directory files = listdir(dir) for file in files: fullname = os.path.join(dir, file) st_values = stat(fullname) # check if its a directory if S_ISDIR(st_values[0]): print file IT-SC book 66 # create directory element self.__fd.write(" self.__fd.write(' name="' + escape(fullname) + '">\n') self.__indexDir(fullname) self.__fd.write(" else: # create regular file entry print dir + file lf = IndexFile(fullname, st_values) self.__fd.write(lf.getXML()) The actual work is just determining those files that are directories and those that are regular files. XML is created accordingly during this process, and written to the output file. When all of the __indexDir calls eventually return, the XML file is closed. Now the program is essentially finished. A helper function named escape is imported from the xml.sax.saxutils module to perform entity substitution against some common characters within XML character data to ensure they do not appear to be markup in the resulting XML. 3.4.1.1 Creating the IndexFile class The IndexFile class is used for an XML representation of file information. This information is derived primarily from the os.stat system call. The class copies information from the stat call into its member variables in its __init__ method, as shown here: def __init__(self, filename, st_vals): """Extract properties from supplied stat object.""" self.filename = filename self.uid = st_vals[4] self.gid = st_vals[5] self.size = st_vals[6] self.accessed = ctime(st_vals[7]) self.modified = ctime(st_vals[8]) self.created = ctime(st_vals[9]) IT-SC book 67 # try for filename extension self.extension = os.path.splitext(filename)[1] In this method, important file information is extracted from the tuple st_vals. This contains the filesystem information returned by the stat call. The __init__ method also tries for a filename extension if possible by checking for the "." character. If you are running Unix, the script tries to use the os.popen function to call the file command, which returns a human-readable description of the content of both text and binary files. It can take much longer to generate, but once in, the XML is valuable and does not need to be regenerated every time we want it: # check contents using file command on linux if os.name == "posix": # Open a process to check file contents fd = popen("file \"" + filename + "\"") self.contents = fd.readline().rstrip( ) fd.close( ) else: # No content information self.contents = self.extension If you're not using Unix, the file command is unavailable, and so the contents information is given the file extension. For example, in a Word file, the XML is """Returns XML version of all data members.""" return (" "\n\t "\n\t "\n\t "\n\t "" + IT-SC book 68 "\n\t "" + "\n\t "\n\t "" + "\n\t "" + "\n") In the preceding code, the XML is thrown together as a series of strings. Another way is to use a DOMImplementation object to create individual elements and insert them into the document's structure (illustrated in Chapter 10). Both of these classes are used to develop a lengthy XML document representing files and metadata for any given section of your filesystem. The complete listing of index.py is shown in Example 3-4 Example 3-4. index.py #!/usr/bin/env python """ index.py usage: python index.py """ import os import sys from os import stat from os import listdir from os import popen from stat import S_ISDIR from time import ctime IT-SC book 69 from xml.sax.saxutils import escape XML_ENC = "ISO-8859-1" """"""""""""""""""""""""""""""""""""""""""""""""""" Class: Index(startingDir, outputFile) """"""""""""""""""""""""""""""""""""""""""""""""""" class Index: """ This class indexes files and builds a resultant XML document. """ def __init__(self, startingDir, outputFile): """ init: sets output file """ self.outputFile = outputFile self.startingDir = startingDir def indexDirectoryFiles(self, dir): """Index a directory structure and creates an XML output file.""" # prepare output XML file self.__fd = open(self.outputFile, "w") self.__fd.write(' XML_ENC + '"?>\n') self.__fd.write(" # do actual indexing self.__indexDir(dir) IT-SC book 70 # close out XML file self.__fd.write("\n") self.__fd.close( ) def __indexDir(self, dir): """Recursive function to do the actual indexing.""" # Create an indexFile for each regular file, # and call the function again for each directory files = listdir(dir) for file in files: fullname = os.path.join(dir, file) st_values = stat(fullname) # check if its a directory if S_ISDIR(st_values[0]): print file # create directory element self.__fd.write(" self.__fd.write(' name="' + escape(fullname) + '">\n') self.__indexDir(fullname) self.__fd.write(" else: # create regular file entry print dir + file lf = IndexFile(fullname, st_values) IT-SC book 71 self.__fd.write(lf.getXML( )) """"""""""""""""""""""""""""""""""""""""""""""""""" Class: IndexFile(filename, stat-tuple) """"""""""""""""""""""""""""""""""""""""""""""""""" class IndexFile: """ Simple file representation object with XML """ def __init__(self, filename, st_vals): """Extract properties from supplied stat object.""" self.filename = filename self.uid = st_vals[4] self.gid = st_vals[5] self.size = st_vals[6] self.accessed = ctime(st_vals[7]) self.modified = ctime(st_vals[8]) self.created = ctime(st_vals[9]) # try for filename extension self.extension = os.path.splitext(filename)[1] # check contents using file command on linux if os.name == "posix": # Open a process to check file # contents fd = popen("file \"" + filename + "\"") self.contents = fd.readline().rstrip( ) IT-SC book 72 fd.close( ) else: # No content information self.contents = self.extension def getXML(self): """Returns XML version of all data members.""" return (" "\n "\n "\n "\n "" + "\n "" + "\n\t "\n\t "" + "\n\t "" + "\n") """"""""""""""""""""""""""""""""""""""""""""""""""" Main """"""""""""""""""""""""""""""""""""""""""""""""""" if __name__ == "__main__": index = Index(sys.argv[1], sys.argv[2]) print "Starting Dir:", index.startingDir IT-SC book 73 print "Output file:", index.outputFile index.indexDirectoryFiles(index.startingDir) 3.4.1.2 Running index.py Running index.py from the command line requires supplying both a starting directory and an XML filename to use as output: $> python index.py /usr/bin/ usrbin.xml The script prints directory names similar to the find command, but after completion, the file usrbin.xml contains something similar to the following: IT-SC book 74 Intel 80386,version 1, dynamically linked (uses shared libs), stripped The XML file's size depends on the particular directory it originated in. By default, the program follows symbolic links (on Unix, symbolic links allow one directory or filename to refer to another), introducing the possibility of forming infinite recursion, so beware! Indexing your home directory or indexing a directory of open source software that you've downloaded is probably the most effective thing to do in this case. 3.4.2 Searching the Index Now that your file data has been abstracted to XML, you can write a SAX event handler to search for items within the file list. SAX is a good choice here, because this document could easily be several megabytes in size, and interpreting it as it is being read is the least resource-intensive approach. The saxfinder.py script takes a single argument (the search text) and parses the supplied XML file checking via its SAX handler interfaces, in order to see if any of the files are of interest to you. The script expects to work on XML as created earlier with index.py. If the contents element of your XML file contains the character data that you supplied on the command line, the file is considered a match and the script prints a message accordingly. If you are running Windows, your contents tags only have the file extension, so your searches are limited to file extensions, unless you alter the code to watch something besides just the contents element. Use three methods of the SAX interface to implement your metadata finder. First, startElement is implemented to both capture the name of the current file element as well as mark when you've entered the character data portion following a contents tag: def startElement(self, name, attrs): if name == "file": self.filename = attrs.get('name', "") elif name == "contents": self.getCont = 1 If you're entering a content element, a flag (self.getCont) is set so that the characters method knows when to gobble up character data and store it in another member variable: IT-SC book 75 def characters(self, ch): if self.getCont: self.contents += ch When an endElement event rolls around, the script examines the contents that have been captured (if any) to see if they match the original command-line parameter. If so, the filename is printed; if not, SAX happily moves on to the next file element within the XML document: def endElement(self, name): if name == "contents": self.getCont = 0 if self.contents.find(self.contentType) > -1: print self.filename, "has", self.contentType, "content." self.contents = "" In addition, the self.getCont flag is disabled after leaving a contents element, to instruct the characters method not to capture data. SAX helps you here by allowing you to process an XML index file that represents an entire filesystem and easily takes up 20 megabytes on your disk. Parsing such a gigantic document with the DOM can be difficult and unbearably slow. Example 3-5 shows the complete listing of saxfinder.py. Example 3-5. saxfinder.py """ saxfinder.py - generates HTML from pyxml.xml """ import sys from xml.sax import make_parser from xml.sax import ContentHandler class FileHandler(ContentHandler): def __init__(self, contentType): self.getCont = 0 IT-SC book 76 self.contents = "" self.filename = "" self.contentType = contentType def startElement(self, name, attrs): if name == "file": self.filename = attrs.get('name', "") elif name == "contents": self.getCont = 1 def characters(self, ch): if self.getCont: self.contents += ch def endElement(self, name): if name == "contents": self.getCont = 0 if self.contents.find(self.contentType) > -1: print self.filename, "has", self.contentType, "content." self.contents = "" # Main fh = FileHandler(sys.argv[1]) parser = make_parser( ) parser.setContentHandler(fh) parser.parse(sys.stdin) IT-SC book 77 You can run saxfinder.py from the command line on both Unix and Windows. You need to supply a search string as the first parameter, and be sure and redirect or pipe an XML document (created with index.py) into standard input: $> python saxfinder.py "C program" < nard.xml The result should be something like this: /home/shm00/nard/xd/server.cpp has C program content. /home/shm00/nard/xd/shmoo.cpp has C program content. /home/shm00/nard/gl-misc/array.cpp has C program content. /home/shm00/nard/gl-misc/vertex.cpp has C program content. /home/shm00/nard/gl-misc/mecogl.cpp has C program content. /home/shm00/nard/gl-misc/drewgl/smugl.cpp has C program content. /home/shm00/nard/gl-misc/drewgl/pal.cpp has C program content. /home/shm00/nard/gl-misc/drewgl/pal.h has C program content. /home/shm00/nard/gl-misc/drewgl/gl.cpp has C program content. 3.5 Building an Image Index If you've ever visited an image library on the Internet, you've probably enjoyed (even taken for granted) the way a collection of small thumbnail images acts as links for full-sized counterparts. Many artists, when presenting a portfolio online, adopt this effective approach to displaying their work. With the rise of digital cameras and scanners, more and more people are finding themselves pulling directories full of images onto the Web in a format that makes for easy browsing. In the next section, we build a Python script that takes a full directory of images and thumbnail images and creates a master HTML page with the thumbnails acting as links to the full-size image. The saxthumbs.py program expects you to have a pre-existing directory of images and thumbnails, and operates on the output of the index.py script we created earlier. In order for the saxthumbs.py SAX handler to correctly process a thumbnail directory, the images need to follow a naming convention (easily changeable by editing the code). Currently, the saxthumbs.py handler expects to find file elements within the XML document that have a corresponding When using index.py to create a list of your image files, point it to a directory that has image files named accordingly: $> ls -l *newimage* -rw-rw-r-- 1 shm00 shm00 98197 Jan 18 11:08 newimage.jpg -rw-rw-r-- 1 shm00 shm00 5272 Jan 18 11:42 t-newimage.jpg IT-SC book 78 In this manner, every file that ends in .jpg and has a corresponding t- 3.5.1 Creating Thumbnail Images There is an easy way to set up your image files on Unix systems, using the convert command. This command is part of the ImageMagick package, and is installed by default by most modern Linux distributions. For other Unix systems, the package is available at http://www.imagemagick.org/. $> convert image.jpg -geometry 192x128 t-image.jpg This will take image.jpg, no matter how large it is, and make a 192x128 size thumbnail in JPEG format. Of course, if the image is a Windows bitmap image (with the .bmp extension), you can do a two-step operation to get JPEG files: $> convert image.bmp image.jpg $> convert image.jpg -geometry 192x128 t-image.jpg Now that you understand how convert works, you can use a simple shell loop to produce small thumbnail images for every .jpg in your image directory: $> for each in *jpg > do > convert $each -geometry 192x128 t-$each > echo $each > done You should end up with the following: -rwxrwxr-x 1 shm00 shm00 97003 Jan 16 22:40 mvc-001s.jpg -rwxrwxr-x 1 shm00 shm00 93373 Jan 16 22:40 mvc-002s.jpg -rwxrwxr-x 1 shm00 shm00 86619 Jan 16 22:40 mvc-003s.jpg -rwxrwxr-x 1 shm00 shm00 94894 Jan 16 22:40 mvc-004s.jpg -rwxrwxr-x 1 shm00 shm00 76210 Jan 16 22:40 mvc-005s.jpg -rwxrwxr-x 1 shm00 shm00 73704 Jan 16 22:40 mvc-006s.jpg -rwxrwxr-x 1 shm00 shm00 80292 Jan 16 22:40 mvc-007s.jpg -rw-rw-r-- 1 shm00 shm00 4434 Jan 21 11:46 t-mvc-001s.jpg -rw-rw-r-- 1 shm00 shm00 4181 Jan 21 11:46 t-mvc-002s.jpg IT-SC book 79 -rw-rw-r-- 1 shm00 shm00 3604 Jan 21 11:46 t-mvc-003s.jpg -rw-rw-r-- 1 shm00 shm00 4634 Jan 21 11:46 t-mvc-004s.jpg -rw-rw-r-- 1 shm00 shm00 3339 Jan 21 11:46 t-mvc-005s.jpg -rw-rw-r-- 1 shm00 shm00 2777 Jan 21 11:46 t-mvc-006s.jpg -rw-rw-r-- 1 shm00 shm00 2996 Jan 21 11:46 t-mvc-007s.jpg This listing represents the convert program applied against mvc-00*.jpg files taken with a digital camera. The saxthumbs.py script produces markup to display both the thumbnails and each individual image. If you run index.py against this directory, you create an XML file that we are able to use a little later in the chapter when we go over the saxthumbs.py process: $> ./index.py /home/shm00/images/ img.xml The new file, img.xml, contains file elements that detail your image files in a format appropriate for the script to manipulate. 3.5.1.1 Creating thumbnails on Windows If you don't have access to Unix (or a scriptable image processor for your operating system), you can create your own image directory on Windows. Just be sure to resize originals and prefix the thumbnails with t-, and make sure that all of the images to be displayed on the Web are in JPEG format (ending with .jpg). For example, if you open the My Pictures directory in Photoshop, you can take each image, resize it to 192x128, and save it as a t- version of its original self. To come back around to our example, once you prepare the directory, you can point index.py at it and generate an XML index file for the images. 3.5.2 Implementing the SAXThumbs Handler The SAXThumbs handler creates an anchor for each thumbnail in the HTML output file, and creates a standalone HTML document to display the full size image. Then, SAXThumbs leaves you with an HTML page showing all of your thumbnails, as well as an HTML page for each full size image. The SAXThumbs handler is implemented as a class inheriting from ContentHandler. The name of the output file, which should contain a preview of all of the thumbnails, is supplied as a command-line parameter and passed to the constructor: def __init__(self, thumbsFilename): self.filename = thumbsFilename def startDocument(self): IT-SC book 80 self.fd = open(self.filename, "w") self.fd.write("Not a Meteor
def endDocument(self):
self.fd.write("\n")
self.fd.close( )
When the end of the XML document is reached, it's assumed that there are no more image files to process, and the thumbnails document is closed.
The rest of the work is done in the startElement method. First, the image name is copied without its path information: def startElement(self, name, attrs):
if name == "file":
filename = attrs.get("name", "")
# pull out just the filename
dir, localname = os.path.split(filename)
localname, ext = os.path.splitext(localname)
Then, the file is examined to determine whether it's a thumbnail or a full image. Thumbnails have HTML anchors around them, which are then added to the thumbnails output file:
if localname.startswith("t-") and ext == ".jpg":
# create anchor tag in thumbs.html
fullImgLink = localname[2:] + ".html"
% (fullImgLink, localname, ext))
If the image is not a thumbnail, but a full image, then a separate HTML file is created that displays the image:
fullImageFile = os.path.join(dir, localname) + ".html"
print "Will create:", fullImageFile
IT-SC book 81 fullImageHTML = ('
\n'% (localname, ext))
lfd = open(fullImageFile, "w")
lfd.write(fullImageHTML)
lfd.close( )
The full-image HTML file is created in the same directory that holds the image. The thumbnails file is created in the same directory from which you're running thumbmaker.py. Example 3-6 shows saxthumbs.py.
Example 3-6. saxthumbs.py from xml.sax import ContentHandler
class SAXThumbs(ContentHandler):
"""
This is the SAX handler that generates a full-
image display (an .html page) for each image file
contained in the XML file.
It also adds an anchor on the thumbs page showing
the thumbnail, and linking to the big image page
that was created first.
"""
def __init__(self, thumbsFilename):
self.filename = thumbsFilename
def startDocument(self):
self.fd = open(self.filename, "w")
self.fd.write("
\n")IT-SC book 82
def endDocument(self):
self.fd.write("\n")
self.fd.close( )
def startElement(self, name, attrs):
if name == "file":
filename = attrs.get("name", "")
# slice out just the filename
dir, localname = os.path.split(filename)
localname, ext = os.path.splitext(localname)
if localname.startswith("t-") and ext == ".jpg":
# create anchor tag in thumbs.html
fullImgLink = localname[2:] + ".html"
% (fullImgLink, localname, ext))
fullImageFile = os.path.join(dir, localname) + ".html"
print "Will create:", fullImageFile
fullImageHTML = ('
\n'% (localname, ext))
lfd = open(fullImageFile, "w")
lfd.write(fullImageHTML)
lfd.close( )
The thumbmaker.py file is a script that loads the XML from standard input, and registers the SAXThumbs class as the chosen handler to use with the SAX parser. Example 3-7 shows thumbmaker.py in its entirety.
IT-SC book 83 Example 3-7. thumbmaker.py
#!/usr/bin/env python
# thumbmaker.py
import sys
from xml.sax import make_parser from saxthumbs import SAXThumbs
# Main
ch = SAXThumbs(sys.argv[1]) parser = make_parser( )
parser.setContentHandler(ch) parser.parse(sys.stdin)
It is interesting to note the similarity to the first script that we wrote in Example 3-2.
To run thumbmaker.py, you first need to make sure you have created the right type of directory containing image files and that you've run index.py across the directory to generate an XML file containing the list of files. Once you have these items, you can pick a name for the index file (such as mythumbs.html) and pass it to the script:
$> python thumbmaker.py mythumbs.html < img.xml
In this case, mythumbs.html is the output file, and the XML source is received from the file img.xml.
3.5.3 Viewing Your Thumbnails
After executing thumbmaker.py, you are left with a thumbnails file that is sitting in your current working directory. You should move this file to the directory that is holding your images:
$> mv mythumbs.html /home/shm00/tw/zero/images/
3.6 Converting XML to HTML
IT-SC book 84 The PyXML package contains XML parsers, including PyExpat, as well as support for SAX and DOM, and much more. While learning the ropes of the PyXML package, it would be nice to have a comprehensive list of all the classes and methods. Since this is a programming book, it seems appropriate to write a Python program to extract the information we need—and in XML, no less!
Let's generate an XML file that details each of the files in the PyXML package, the classes therein, and the methods of the class. This process allows us to generate quick, usable XML. Rather than a replacement for all the snazzy code-to-documentation generators out there, Example 3-8 shows a simple, quick way to generate XML that we can experiment with and use throughout the examples in this chapter. After all, when manipulating XML, it helps to have a few hundred thousand bytes of it sitting around to play with. (This program also demonstrates the simplicity of examining all the files in a directory tree in using the os.path.walk function.)
Example 3-8. genxml.py
""" genxml.py
Descends PyXML tree, indexing source files and creating
XML tags for use in navigating the source.
"""
import os import sys
from xml.sax.saxutils import escape
def process(filename, fp):
print "* Processing:", filename,
# parse the file
pyFile = open(filename)
fp.write("
IT-SC book 85 inClass = 0
line = pyFile.readline( )
while line:
line = line.strip( )
if line.startswith("class") and line[-1] == ":":
if inClass:
fp.write(" \n")
inClass = 1
fp.write("
elif line.find("def") > 0 and line[:-1] == ":" and inClass:
fp.write("
line = pyFile.readline( )
pyFile.close( )
if inClass:
fp.write(" \n")
inClass = 0
fp.write("\n")
def finder(fp, dirname, names):
"""Add files in the directory dirname to a list."""
for name in names:
if name.endswith(".py"):
path = os.path.join(dirname, name)
if os.path.isfile(path):
IT-SC book 86 process(path, fp)
def main( ):
print "[genxml.py started]"
xmlFd = open("pyxml.xml", "w")
xmlFd.write("\n")
xmlFd.write("
os.path.walk(sys.argv[1], finder, xmlFd)
xmlFd.write("")
xmlFd.close( )
print "[genxml.py finished]"
if __name__ == "__main__":
main( )
The main function in Example 3-8 uses the os.path.walk function to search your PyXML directory for Python files. For each Python source file that exists beneath the starting directory (the argument to the script), the process function is called to extract class information. That function writes the extracted information into the open XML file.
At this point, the script proceeds to parse each Python source file, highlighting each of the classes and methods contained within them by parsing each line for relevant keywords such as class and def: def process(filename, fp):
print "* Processing:", filename,
# parse the file
pyFile = open(filename)
IT-SC book 87 fp.write("
inClass = 0
line = pyFile.readline()
while line:
line = line.strip()
When the program finds a class declaration, it creates the appropriate class tag and attributes within the XML document:
if line.startswith("class") and line[-1] == ":":
if inClass:
fp.write(" \n")
inClass = 1
fp.write("
When the program encounters a method definition, it replaces special characters with entities so they don't cause problems in the XML. The method definition string is trimmed, and then surrounded with the appropriate markup:
elif line.find("def") > 0 and line[:-1] == ":" and inClass:
fp.write("
line = pyFile.readline()
After a file is complete, the program closes out the last class it was in, if any, and closes out the file tag as well:
pyFile.close()
if inClass:
fp.write(" \n")
inClass = 0
fp.write("\n")
Python simplifies the work of parsing the text. Each line is manipulated quite a bit, quotation marks are escaped with entities (using the escape function from the xml.sax.saxutils module), and XML tags are placed around class definitions and method names.
IT-SC book 88 To run this program from the shell:
$> python genxml.py /home/chris/PyXML/xml
The parameter to the script is the path to your PyXML source directory (including the xml subdirectory).
3.6.1 The Generated Document
The XML that is generated is placed in a file called pyxml.xml. Each file element looks something like this:
IT-SC book 89
Note that the name attribute of the file tag varies depending upon what your parameter is to the script (your PyXML source path). Functions not defined as methods in a class are not included by the simple parsing loop (hey, this isn't a compiler!), but you should be aware that the XML support provided by both the standard library and the PyXML package includes many useful functions—read the reference documentation for more information on those. The escape function we use in this script is a perfect example of this. If you're new to Python, you'll find that little helper functions are characteristic of Python libraries; most of the small utilities needed to make the larger facilities easier to use have already been included, allowing you to concentrate on your application.
If you spend some time reviewing this XML file, you will start to become familiar with the scope of the PyXML toolkit. A script is provided a little later in this chapter that converts this XML to HTML using the SAX API and Python string manipulation features. Figure 3-2 shows the XML within a browser.
Figure 3-2. genhtml.py output in a browser
IT-SC book 90
3.6.2 The Conversion Handler
You can finish off this program by implementing the PyXMLConversionHandler class. This class generates HTML from the XML file we created earlier. The process allows you to load the HTML file into your browser and see all of the files, classes, and methods within PyXML in formatted text. Create this class, as shown in Example 3-9, in the file handlers.py.
Example 3-9. handlers.py from xml.sax import ContentHandler
class PyXMLConversionHandler(ContentHandler):
"""A simple handler implementing 3 methods of
the SAX interface."""
def __init__(self, fp):
IT-SC book 91 """Save the file object that we generate HTML into."""
self.fp = fp
def startDocument(self):
"""Write out the start of the HTML document."""
self.fp.write("
\n")def startElement(self, name, attrs):
if name == "file":
# generate start of HTML
s = attrs.get('name', "")
self.fp.write("
File: %s
\n" % s)
print "* Processing:", s
elif name == "class":
self.fp.write(" " * 3 + "Class: "
+ attrs.get('name', "") + "
\n")
elif name == "method":
self.fp.write(" " * 6 + "Method: "
+ attrs.get('name', "") + "
\n")
def endDocument(self):
"""End the HTML document we're generating."""
self.fp.write("")
While the conversion itself is very straightforward, one interesting thing to note is that this class writes its output to a file object passed to the constructor instead of building a string of XML text in memory. This avoids storing a potentially large buffer in memory and building it incrementally with many memory copies. If the string is required to be in memory when the process is complete, the creator can provide a StringIO instance as the file to write to; the StringIO
IT-SC book 92 implementation is more efficient at building a large string than many string concatenations. This is a Python idiom that has proven its utility over a wide range of projects.
3.6.3 Driving the Conversion Handler
The main script really isn't any different from the others we've looked at so far. We create the parser and instantiate our handler class, register the handler, and set the parser in motion. This process is shown in Example 3-10.
Example 3-10. genhtml.py
#!/usr/bin/env python
#
# generates HTML from pyxml.xml
import sys
from xml.sax import make_parser from handlers import PyXMLConversionHandler
dh = PyXMLConversionHandler(sys.stdout) parser = make_parser( )
parser.setContentHandler(dh) parser.parse(sys.stdin)
The output from this script is written to the standard output stream.
3.7 Advanced Parser Factory Usage
PyXML features several parsers, and multiple ways to instantiate them, depending on whether you're using SAX, trying to create a DOM tree, or doing something completely different. Designed for portable code, a ParserFactory class is provided that supplies a SAX-ready parser guaranteed available in your runtime environment. Additionally, you can explicitly create a parser (or SAX driver) by dipping into any specific package, such as PyExpat. We illustrate an example of both, but normally you should rely on the parser factory to instantiate a parser.
The make_parser function (imported from xml.sax) returns a SAX driver for the first available parser in the list that you supply, or returns an available parser if no list is specified or if
IT-SC book 93 the list contains parsers that are not found or cannot be loaded. The make_parser function has its roots as part of the xml.sax.saxexts.ParserFactory class, but it is better to import the method from xml.sax (more on this in a bit). For example: from xml.sax import make_parser parser = make_parser( )
At the time of this writing, if you have PyXML installed, a call to make_parser without an argument is sure to return either a PyExpat or xmlproc driver. If you dig into the source of the xml.sax module, you will see this list supplied to the ParserFactory class. If you instantiate a parser factory directly out of xml.sax.saxexts, you need to be sure to supply a list containing the name of at least one valid parser, or it won't be able to create a parser:
>>> from xml.sax.saxexts import ParserFactory
>>> p = ParserFactory( )
>>> parser = p.make_parser( )
Traceback (most recent call last):
File "
File "/usr/local/lib/python2.0/site-packages/_xmlplus/sax/saxexts.py",
line 77, in make_parser
raise SAXReaderNotAvailable("No parsers found", None) xml.sax._exceptions.SAXReaderNotAvailable: No parsers found
If you supply a list of parsers or drivers, you get what you're after:
>>> from xml.sax.saxexts import ParserFactory
>>> p = ParserFactory(["xml.sax.drivers.drv_pyexpat"])
>>> parser = p.make_parser( )
In most cases, it's a good idea to use the make_parser function from xml.sax, but it's also valuable to know what is going on under the hood. Several factory classes are available, with variations for HTML, SGML, non-validating XML, and validating XML parsers.
3.8 Native Parser Interfaces
Now that we've looked at how SAX can be used and have seen just how regular the code is to set up the parser and the ContentHandler, you may be wondering how much of that ease comes from using SAX and how much is a matter of convenience functions in the Python libraries. While we won't delve deeply into the native interfaces of the individual parsers, this is a good question, and can lead to some interesting observations.
IT-SC book 94 The key advantage to using SAX is that the callback methods have the same names and significance regardless of the actual parser you use. There are at least two nice results of this: changing parsers does not affect your application, and your code is more maintainable because someone new to the code is more likely to know the SAX interface than any particular parser-specific interface.
So just how do the native interfaces to the individual parsers differ from SAX, and why would we choose to use them instead? Let's take a quick look at the PyExpat parser to get a taste of the differences.
3.8.1 Using PyExpat Directly
Of course, to use PyExpat, you need to have it installed. It is included as part of the Python installer for Windows, and is built automatically on Unix if you have the Expat library installed. If you did not install PyExpat as part of Python, it is installed as part of the PyXML package.
PyExpat resides in the xml.parsers.expat module. If we want to modify our last example to use PyExpat directly, we don't have a lot of work to do, but there are a few changes. Since the PyExpat handler methods closely match the SAX handlers, at least for the basic use we demonstrate here, we can use the same handler class we've already written. The imports won't need to change much:
#!/usr/bin/env python
import sys
from xml.parsers import expat from handlers import PyXMLConversionHandler
Once the parser is imported, it can be created and used: parser = expat.ParserCreate( )
Were we to do this at the interactive prompt, we could poke at the parser object to see what attributes it has:
>>> from xml.parsers import expat
>>> parser = expat.ParserCreate( )
>>> dir(parser)
['CharacterDataHandler', 'CommentHandler', 'DefaultHandler', 'DefaultHandlerExpa nd', 'EndCdataSectionHandler', 'EndElementHandler', 'EndNamespaceDeclHandler', '
IT-SC book 95 ErrorByteIndex', 'ErrorCode', 'ErrorColumnNumber', 'ErrorLineNumber', 'ExternalE ntityParserCreate', 'ExternalEntityRefHandler', 'GetBase', 'NotStandaloneHandler
', 'NotationDeclHandler', 'Parse', 'ParseFile', 'ProcessingInstructionHandler',
'SetBase', 'StartCdataSectionHandler', 'StartElementHandler', 'StartNamespaceDec lHandler', 'UnparsedEntityDeclHandler', 'ordered_attributes', 'returns_unicode',
'specified_attributes']
That certainly doesn't look like a SAX parser!
There is no setContentHandler method, nor is there anything that takes its place. To register our content handler, we need to set various attributes to the methods of a content handler instance: dh = PyXMLConversionHandler(sys.stdout) parser.StartElementHandler = dh.startElement parser.EndElementHandler = dh.endElement parser.CharacterDataHandler = dh.characters
This isn't hard, but it is certainly more tedious than the SAX setContentHandler method, and the code actually needs to be changed, as we need to use more methods on the handler object.
Once we've initialized the handler methods we're interested in using, we can start the parse. Again, this is a little different from the SAX version: parser.Parse(sys.stdin.read( ), 1)
We know what sys.stdin.read( ) does, but the 1 used for the second parameter looks suspiciously like a magic number in our source code. It is actually a Boolean value indicating that the string being passed to the Parse method is the final chunk of the input; Parse can be called multiple times with smaller portions of the input and the flag set to 0, and then called with an indicator of 1 for the final chunk of data. This can be useful when reading data asynchronously from a network connection.
When parsing XML from a file object, the following method is also available: parser.ParseFile(sys.stdin)
The complete script that uses the handler with PyExpat is shown in Example 3-11.
Example 3-11. genhtml2.py with PyExpat
IT-SC book 96 """ genhtml2.py - generates HTML from pyxml.xml
"""
import sys
from xml.parsers import expat from handlers import PyXMLConversionHandler
dh = PyXMLConversionHandler(sys.stdout) parser = expat.ParserCreate( )
parser.StartElementHandler = dh.startElement parser.EndElementHandler = dh.endElement parser.CharacterDataHandler = dh.characters parser.ParseFile(sys.stdin)
The output is to the standard output stream. If opened in your browser, it shows you all of the classes of the PyXML package and their methods, exactly as the pure SAX version of this example did.
IT-SC book 97 Chapter 4. The Document Object Model
The Document Object Model (DOM) is an interface that exposes document structure programmatically to developers. Perhaps the most common application of the DOM is "Dynamic HTML" (DHTML), where an HTML document can be modified programmatically within the browser using an embedded scripting language. Typically, the scripting language is some flavor of ECMAScript (such as JavaScript or JScript), since most browsers support it, but others can be used as well. (For browsers on Windows, this can even be Python!) This allows you to change the background color of a table cell, or dynamically change font faces after the page is in the browser. The DOM defines the interface for vendors to offer compatible APIs.
The DOM is also extremely useful when exposed by a library such as the Python Standard Library or PyXML. It can allow you to use Python to manipulate an XML document already in memory. With the DOM interfaces, you can either change or extract portions of the document.
4.1 The DOM Specifications
The Document Object Model is defined in a series of recommendations from the W3C. The specifications clearly cover XML (or we would not be describing them in this book), but they cover other things as well. The initial version of the DOM actually came from the HTML world; browser vendors invented it in various flavors as part of the APIs available to client-side scripts embedded in web pages. Since the vendors each implemented different interfaces, there was a call from content creators to have a standardized interface so their pages would work in at least roughly equivalent ways on the different browsers. Since the W3C is the best available shared ground on which the vendors could build a common specification, the DOM specifications are developed there.
All standards organizations face issues regarding the longevity of their specifications, and the W3C is no exception, no matter that it is quite young compared to more traditional standards groups such as ANSI and ISO. Given the relative youth of the W3C, it has had to deal with these issues almost from the start due to the rapid pace of development and the way standards are applied on the Internet. It does follow a traditional model however, rather than following the less formal (though highly effective) model of the Internet Engineering Task Force (IETF).
Most of the W3C recommendations provide a version number of the major.minor style favored by software developers, perhaps due to the origins of the organization. This is probably most prominent when we look at the HTML specifications; many versions have been released, and each is distinct from the others. Documents that contain anything beyond the simplest content cannot hope to comply with more than one version of the recommendation. This seems difficult to avoid for a markup language, but the effect is often that the standards are not as valuable as they could be if it were possible to maintain a higher degree of version independence.
The W3C is doing something different with the DOM. The specifications for the DOM do not have versions in the same sense that the HTML specification does. The new versioning model is also being used with the Cascading Style Sheets (CSS) recommendations, although those specifications are outside the scope of this book.
IT-SC book 98 The DOM specification has been developed as a family of individual specifications, and the family can be described along two different axes: breadth and depth. When we think about the breadth of the recommendations, we can describe a broad family as including many features. For depth, we can describe a deep family as reaching further into the details as well as covering basic functionality. A broad family does many things, while a deep family tree covers many details. The W3C describes functional areas as features, while it describes depth of detail as levels.
4.1.1 Levels of the Specification
The levels of a specification are interesting to discuss first because they can be most confusing for many people. It is common to hear levels described as being just a strange name for the traditional notion of versions, but they are quite different. (Unfortunately, the DOM specifications themselves are not always clear about this.) As each feature of the DOM is enhanced, new levels are defined. This similarity to traditional versions certainly makes it easy to confuse the two concepts, but there is an important difference: an implementation of the second level must include an implementation of the first; advancing beyond the first level does not break compatibility for code that only expects to work with the older variant of the specification.
Each level of the DOM specifications cuts across the entire breadth of the DOM family, as it existed when it was defined (with one exception). Successive levels have introduced new features as well. Implementations are not required to implement all the features of the DOM, but generally need to implement the features they include at the same level.
At the time of this writing, two levels of the DOM have been defined by W3C recommendations, with a third level being developed by working groups within the W3C. The primordial interfaces defined by browser implementations before the DOM standardization began are often described as "Level 0." The Level 1 specification from the W3C consists of a single recommendation that defines only two features, Core and HTML. This level provides general support for HTML and the XML 1.0 recommendation, but nothing else. For Level 2, the W3C broke the specification into six different documents. The Core feature was split into the Core and XML features, and support for Namespaces was added. New features added in Level 2 include an events model (mostly, but not entirely, for use in browsers), Cascading Style Sheets, document traversal, range specifications, and a vague concept of document views. Oddly, the HTML feature for Level 2 has not been completed and there has been no visible progress for quite some time.
The third level of the DOM, still only available as a set of working drafts, contains just four documents at this time. The Core, XML, and Events features are further refined, but most of the interesting work is taking place in new features. The current plans include new features for schemas (supporting at least the DTD and XML Schema languages), loading and saving XML documents, and an object model for XPath expressions. (We look at XPath in the next chapter, but it's too early to consider that the XPath feature of the DOM is ready.)
4.1.2 Feature Specifications
The features defined by the DOM vary from level to level, with new features being added and old features being split into separate features. The former is not a problem because code that works with an implementation of earlier levels simply will not need the newer features. In
IT-SC book 99 practice, the second has not been demonstrated to introduce any difficulty either, if only because Level 2 implementations always implement both the Core and XML features. For any implementation, the only required feature is the Core.
Since most Python implementations of the DOM provide at least some features from Level 2, and Level 3 exists only in draft form, let's take a look at what each feature defined for Level 2 provides to the application developer.
Core
This includes basic structures required to expose well-formed XML documents without exposing any DTD information. In particular, entities, notations, entity references, and processing instructions are not provided. These are the interfaces with which we are concerned with in this chapter.
XML
This feature set adds additional interfaces used to represent entity and notation declarations provided by the document type declaration (though not the document type itself), and some lexical information helpful in generating a modified document, including CDATA sections, entity references, and processing instructions.
Events
This feature is interesting in that it is broken down into several specific subfeatures. All implementations that support any type of events must support the basic Events feature, but only need to support the specific subfeatures which make sense for the implementation. The subfeatures include support for various classes of user-interface events and document- modification events.
Range
The range feature provides interfaces that make it possible to describe a portion of a document that cannot be represented as a sequence of nodes; this can be especially useful when describing a selection from the document as might be highlighted by the user.
Traversal
This provides support for traversal over the nodes of a document (or part of a document) in either the order in which they are found in the document, or as a tree-based traversal where the application guides a cursor to visit child nodes, parent nodes, or siblings during the traversal. Nodes can be filtered so the application need not deal with nodes it is not interested in.
Views
A vague specification that deals with providing multiple types of views on a document. This is not clearly useful.
StyleSheets
IT-SC book 100 An abstract interface used to represent stylesheets. This is not specifically bound to Cascading Style Sheets, but may be used to represent other kinds of stylesheets as well. Since each stylesheet language is substantially different, this does not provide much styling information.
CSS
The CSS feature includes extensions of the Style Sheets interfaces that provide substantially more style information. These interfaces provide a great deal of information about CSS Level 2 stylesheets. This is intended to be used in browsers and editors, which are expected to update their presentation based on changes to the stylesheets using these interfaces.
Additional DOM features are being prepared outside the DOM working group for specific XML- based languages. Information about these and the specifications from the DOM working group is available online at http://www.w3.org/DOM/DOMTR.
4.2 Understanding the DOM
The DOM structure is essentially a hierarchy of node objects. Beginning with the root of the document (not the same as the document element), all constructs in the document are represented by nodes of various types, whether an element, text, attributes of elements, or other less common node types. Each node contains a list of references to child nodes, which can in turn be of the same types as those contained by the parent node. Therefore, a complete document looks just like a tree, all the way from the "trunk" (or root element of the tree) out to the leaf nodes representing text, childless elements, comments, processing instructions, and possibly other constructs. Figure 4-1 shows a very simple DOM hierarchy including a root element, two child elements, and their respective child text elements. Usually the character data of an element consists of multiple text nodes depending on the parser in use. Contiguous strings of textual data become sequences of text nodes.
Figure 4-1. A simple DOM hierarchy
When a document is represented by the DOM, an object hierarchy represents the entire document. As with other nodes, it can contain children; the outermost element of the document is simply a child of the document node. The document can have other children; comments and processing instructions can precede or succeed the
IT-SC book 101 document element and appear in the proper order as children of the document. The document type declaration is also represented as a child of the document.
The W3C was careful to specify the DOM in a language-independent way, and each programming language has its own way to present the interfaces to the application programmer; each of these mappings of the DOM into the idioms of the target language is called a binding of the DOM. The W3C includes bindings for Java and ECMAScript as part of the DOM specifications. For Python, the official source of the DOM binding is the Python XML-SIG. The binding developed by the SIG members has been documented in the Python Library Reference, which is part of the standard documentation package for Python. Reference material for the DOM has been included in Appendix D of this book, but the standard documentation should be considered the authoritative document for this binding.
The DOM specifications provide the interfaces as CORBA IDL modules and Java interfaces, but does not specify (or even recommend) that the language-specific IDL mappings adopted by the Object Management Group (OMG) be used. In fact, the Java interfaces provided by the W3C do not match the IDL-to-Java mapping. For Python, the XML-SIG decided that a somewhat more Python-friendly mapping would be used, with some concessions made to the IDL-to-Python mapping. Since no one seems to be using the IDL-derived form of the binding, we cover only the Python- centric version of the DOM binding in this book.
4.3 Python DOM Offerings
Python has several different ways for working with the DOM. The one you choose should best fit your needs. minidom is smaller and faster than a fully compliant DOM, but suits the needs of most users. pulldom provides a way to build only the portion of the DOM needed for a particular application, allowing the DOM to be more easily used when working with large documents or tight memory constraints. 4DOM is a full-fledged DOM Level 2 implementation. While these are the dominant implementations of the DOM for Python, and the only implementations described here, realize that there are additional implementations available that may be more tailored to your requirements.
4.3.1 Streamlining with Minidom minidom , part of the xml.dom package included with both the Python standard library and PyXML, is a lightweight DOM implementation. Its goal is to provide a simple implementation and smaller memory footprint than a full DOM implementation. The methods for creating the DOM are simple as well. minidom also supports functions for working with string-length XML chunks and methods for extracting them.
Overall, minidom may be best for loading simple (not necessarily small) configuration files for your applications, dealing with form submissions from web pages, handling user authorization, and using it anywhere a "little" bit of XML is needed. You can reduce memory and time overhead by using minidom. These are two elements of significant importance in web application development.
4.3.2 Using Pulldom
IT-SC book 102 pulldom , which also may be imported from the xml.dom package, may be just the thing to save your life when faced with the task of taking a portion of a large XML document and creating a DOM instance of the subset for manipulation. pulldom essentially allows for the construction of selected portions of a DOM based on SAX events. The module uses minidom for the actual nodes it returns. pulldom seeks to be a middle ground between the DOM and SAX. pulldom wants to overcome the state-management (the place-marking mentioned earlier) of SAX, but also preserve its stream- based processing for speed and efficiency. pulldom also seeks to simplify the self-similar, intricately complex nature of a complete DOM tree, its many nodes and lists, and its memory- gobbling nature.
4.3.3 4DOM: A Full Implementation
Both minidom and pulldom have their specific fits, but for the remainder of this book, we work with 4DOM. This is a DOM implementation that implements most of the Level 2 features that actually make sense outside a browser.
After your experience with SAX earlier in this chapter, interacting with the DOM may seem incredibly easy by comparison. However, dealing with a seemingly endless intricacy of stacked node classes may send you running back to SAX to do your string comparisons. However you fare, the next sections seek to introduce you to working with the DOM in Python, and to provide a reference to its interfaces.
Regardless of the implementation you use, there are two basic types of operations you can perform with the DOM. The most common operations involve retrieving information from the document, which we discuss first. Once we cover that, we move on to explain how to use the DOM to modify and create documents.
4.4 Retrieving Information
Retrieving information from a document is easy using the DOM. Most of the work lies in traversing the document tree and selecting the nodes that are actually interesting for the application. Once that is done, it is usually trivial to call a method of the node (or nodes), or to retrieve the value of an attribute of the node. In order to extract information using the DOM, however, we first need to get a DOM document object.
4.4.1 Getting a Document Object
Perhaps the most glaring hole in the DOM specifications is that there is no facility in the API for retrieving a document object from an existing XML document. In a browser, the document is completely loaded before the DOM client code in the embedded or linked scripts can get to the document, so the document object is placed in a well-known location in the script's execution environment. For applications that do not live in a web browser, this approach simply does not work, so we need another solution.
Our solution depends on the particular DOM implementation we use. We can always create a document object from a file, a string, or a URL.
IT-SC book 103 4.4.1.1 Loading a document using 4DOM
Creating a DOM instance to work with is easy in Python. Using 4DOM, we need call only one function to load a document from an open file: from xml.dom.ext.reader.Sax2 import FromXmlStream doc = FromXmlStream(sys.stdin)
4.4.1.2 Loading a document using minidom
There are two convenient functions in the xml.dom.minidom module that can be used to load a document. The parse function takes a parameter that can be a string containing a filename or URL, or it can be a file object open for reading: import xml.dom.minidom doc = xml.dom.minidom.parse(sys.stdin)
Another function, parseString, can be used to load a document from a buffer containing XML text that has already been loaded into memory: doc = xml.dom.minidom.parseString("
4.4.2 Determining a Node's Type
You can use the constants built in to the DOM to see what type of node you are dealing with. It may be an element, an attribute, a CDATA section, or a host of other things. (All the node type constants are listed in Appendix D.)
To test a node's type, compare its nodeType attribute to the particular constant you're looking for. For example, a CDATASection instance has a nodeType equal to CDATA_SECTION_NODE. An Element (with potential children) has a nodeType equal to ELEMENT_NODE. When traversing a DOM tree, you can test a node at any point to determine whether it is what you're looking for: for node in nodes.childNodes:
if node.nodeType == node.ELEMENT_NODE:
print "Found it!"
The Node interface has other identifying properties, such as its value and name. The nodeName value represents the tag name for elements, while in a text node the nodeName is simply #text. The nodeValue attribute may be null for elements, and should be the actual character data of a text element or other leaf-type element.
4.4.3 Getting a Node's Children
When dealing with a DOM tree, you primarily use nodes and node lists. A node list is a collection of nodes. Any level of an XML document can be represented as a node list. Each node in the list
IT-SC book 104 can in turn contain other node lists, representing the potential for infinite complexity of an XML document.
The Node interface features two methods for quickly getting to a specific child node, as well as a method to get a node list containing a node's children. firstChild refers to the first child node of any given node. The interface shows None if the node has no children. This is handy when you know exactly the structure of the document you're dealing with. If you are working with a strict content model enforced by a schema or DTD, you may be able to count on the fact that the document is organized in a certain way (provided you included a validation step). But for the most part, it's best to leverage the spirit of XML and actually traverse the document for the data you're looking for, rather than assume there is logic to the location of the data. Regardless, firstChild can be very powerful, and is often used to retrieve the first element beneath a document element.
The lastChild attribute is similar to firstChild, but returns the last child node of any given node. Again, this can be handy if you know the exact structure of the document you're working with, or if you're trying to just get the last child regardless of the significance of that child.
The childNodes attribute contains a node list containing all the children of the given node. This attribute is used frequently when working with the DOM. When iterating over children of an element, the childNodes attributes can be used for simple iteration in the same way that you would iterate over a list: for child in node.childNodes:
print "Child:", child.nodeName
The value of the childNodes attribute is a NodeList object. For the purpose of retrieving information from the DOM, it behaves like a Python list, but does not support "slicing." NodeList objects should not be used to modify the content of the DOM as the specific behaviors may differ among DOM implementations.
The NodeList interface features some additional interfaces beyond those provided by lists. These are not commonly used with Python, but are available since the DOM specifies their presence and behavior. The length attribute indicates the number of nodes in the list. Note that the length returns the total number, but that indexing begins at zero. For example, a NodeList with a length of 3 has nodes at indices 0, 1, and 2 (which mirrors the way an array is normally indexed in Python). Most Python programmers prefer to use the len built-in function, which works properly with NodeList objects.
The item method returns the item at the specific index passed in as a parameter. For example, item(1) returns the second node in the NodeList, or None if there are fewer than two nodes. This is distinct from the Python indexing operation, for which a NodeList raises IndexError for an index that is out of bounds.
4.4.4 Getting a Node's Siblings
Since XML documents are hierarchical and the DOM exposes them as a tree, it is reasonable to want to get the siblings of a node as well as its children. This is done using the previousSibling and nextSibling attributes. If a node is the first child of its parent, its
IT-SC book 105 previousSibling is None; likewise, if it is the last child, its nextSibling is None. If a node is the only child of its parent, both of these attributes are None, as expected.
When combined with the firstChild or lastChild attributes, the sibling attributes can be used to iterate over an element's children. The required code is slightly more verbose, but is also better suited for use when the document tree is being modified in certain ways, especially when nodes are being added to or removed from the element whose children are being iterated over.
For example, consider how Directory elements could be removed from another Directory element to leave us with a Directory containing only files. If we iterate over the top element using its childNodes attribute and remove child Directory elements as we see them, some nodes are not properly examined. (This happens because Python's for loops use the index into the list, but we're also shifting remaining children to the left when we remove one, so it is skipped as the loop advances.) There are many ways to avoid skipping elements, but perhaps the simplest is to use nextSibling to iterate: child = node.firstChild while child is not None:
next = child.nextSibling
if (child.nodeType == node.ELEMENT_NODE
and child.tagName == "Directory"):
node.removeChild(child)
child = next
4.4.5 Extracting Elements by Name
The DOM can provide some advantages over SAX, depending on what you're trying to do. For starters, when using the DOM, you don't have to write a separate handler for each type of event, or set flags to group events together as was done earlier with SAX in Example 3-3. Imagine that you have a long record of purchase orders stacked up in XML. Someone has approached you about pulling part numbers, and only part numbers, out of the document for reporting purposes. With SAX, you can write a handler to look for elements with the name used to identify part numbers (sku in the example), and then set a flag to gobble up character events until the parser leaves the part number element. With the DOM, you have a different approach using the getElementsByTagName method of the Document interface.
To show how easy this can make some operations, let's look at a simple example. Create a new XML file as shown in Example 4-1, po.xml. This document is the sample purchase order for the next script:
Example 4-1. po.xml
IT-SC book 106
Using the DOM, you can easily create a list of nodes that references all nodes of a single element type within the document. For example, you could pull all of the sku elements from the document into a new list of nodes. This list can be used like any other NodeList object, with the difference that the nodes in the list may not share a single parent, as is the case with the childNodes value. Since the DOM works with the structural tree of the XML document, it is able to provide a simple method call to pull a subset of the document out into a separate node list. In Example 4-2, the getElementsByTagName method is used to create a single NodeList of all the sku elements within the document. Our example shows that sku elements have text nodes as children, but we know that a string of text in the document may be presented in the DOM as multiple text nodes. To make the tree easier to work with, you can use the normalize method of the Node interface to convert all adjacent text nodes into a single text node, making it easy to use the firstChild attribute of the Element class to retrieve the complete text value of the sku elements reliably.
Example 4-2. po.py
IT-SC book 107 #!/usr/bin/env python
from xml.dom.ext.reader.Sax2 import FromXmlStream import sys
doc = FromXmlStream(sys.stdin)
for sku in doc.getElementsByTagName("sku"):
sku.normalize( )
print "Sku: " + sku.firstChild.data
Example 4-2 requires considerably less code than what is required if you are implementing a SAX handler for the same task. The extraction can operate independently of other tasks that work with the document. When you run the program, again using po.xml, you receive something similar to the following on standard output:
Sku: 229-987488
Sku: 228-988347
Sku: 221-388833
You can see something similar being done using SAX in Example 3-3.
4.4.6 Examining NodeList Members
Let's look at a program that puts many of these concepts together, and uses the article.xml file from the previous chapter (Example 3-1). Example 4-3 shows a recursive function used to extract text from a document's elements.
Example 4-3. textme.py
#!/usr/bin/env python
from xml.dom.ext.reader.Sax2 import FromXmlStream import sys
def findTextNodes(nodeList):
IT-SC book 108 for subnode in nodeList:
if subnode.nodeType == subnode.ELEMENT_NODE:
print "element node: " + subnode.tagName
# call function again to get children
findTextNodes(subnode.childNodes)
elif subnode.nodeType == subnode.TEXT_NODE:
print "text node: ",
print subnode.data
doc = FromXmlStream(sys.stdin) findTextNodes(doc.childNodes)
You can run this script passing article.xml as standard input:
$> python textme.py < article.xml
It should produce output similar to the following: element node: webArticle text node:
element node: header text node:
element node: body text node: Seattle, WA - Today an anonymous individual
announced that NASA has completed building a
Warp Drive and has parked a ship that uses
the drive in his back yard. This individual
claims that although he hasn't been contacted by
IT-SC book 109 NASA concerning the parked space vessel, he assumes
that he will be launching it later this week to
mount an exhibition to the Andromeda Galaxy.
text node:
You can see in the output how whitespace is treated as its own text node, and how contiguous strings of character data are kept together as text nodes as well. The exact output you see may vary from that presented here. Depending on the specific parser you use (consider different versions or different platforms as different parsers since the buffering interactions with the operating system can be relevant), the specific boundaries of text nodes may differ, and you may see contiguous blocks of character data presented as more than one text node.
4.4.7 Looking at Attributes
Now that we've seen how to examine the hierarchical content of an XML document using the DOM, we need to take a look at how we can use the DOM to retrieve XML's only nonhierarchical component: attributes. As with all other information in the DOM, attributes are described as nodes. Attribute nodes have a very special relationship with the tree structure of an XML document; we find that the interfaces that allow us to work with them are different as well.
When we looked at the child nodes of elements earlier (as in Example 4-3), we only saw nodes for child elements and textual data. From this, we can reasonably surmise that attributes are not children of the element on which they are included. They are available, however, using some methods specific to Element nodes. There is an attribute of the Node interface that is used only for attributes of elements.
The easiest way to get the value of an attribute is to use the getAttribute method of the element node. This method takes the name of the attribute as a string and returns a string giving the value of the attribute, or an empty string if the attribute is not present. To retrieve the node object for the attribute, use the getAttributeNode method instead; if the attribute does not exist, it returns None. If you need to test for the presence of an attribute without retrieving the node or attribute value, the hasAttribute method will prove useful.
Another way to look at attributes is using a structure called a NamedNodeMap. This object is similar in function to a dictionary, and the Python version of this structure shares much of the interface of a dictionary. The Node interface includes an attribute named attributes that is only used for element nodes; it is always set to None for other node types. While the NamedNodeMap supports the item method and length attribute much as the NodeList interface does, the normal way of using it in Python is as a mapping object, which supports most of the interfaces provided by dictionary objects. The keys are the attribute names and the values are the attribute nodes.
4.5 Changing Documents
IT-SC book 110 Now that we've looked at how we can extract information from our documents using the DOM, we probably want to be able to change them. There are really just a few things we need to know to make changes, so we describe the basic operations and then show a few examples. The basic operations involved in modifying a document center around creating new nodes, adding, moving, and removing nodes, and modifying the contents of nodes. Since we often want to add new elements and textual content, we start by looking at creating new nodes.
4.5.1 Creating New Nodes
Most of the time, new nodes need to be created explicitly. Since the DOM is defined as a set of interfaces rather than as concrete classes, the only way to create new nodes is to make call methods on the objects we already have in hand. Fortunately, the Document interface includes a large selection of factory methods we can use to create new nodes of most types. (Methods for creating entity and notation nodes are noticeably absent, but most applications should not find themselves constrained by that.)
The most used of these factory methods are very simple, and are used to create new element and text nodes. For elements, use the createElement method, with the tag name of the element to create as the only parameter. Text nodes can be created using the createTextNode method, passing the text of the new node as the parameter. For the details on the other node factory methods, see the reference material in Appendix D.
4.5.2 Adding and Moving Nodes
There are some very handy methods available for moving nodes to different locations on the tree. These methods appear on the basic Node interface, so all DOM nodes provide these. There are constraints on the use of these nodes: you cannot use them to construct documents which do not make sense structurally, and well-formedness of the document is ensured at all times. For example, an exception is raised if you attempt to add a child to a text node, or if you try to add a second child element to the document object. appendChild( newChild)
Takes a newChild node argument and appends it to the end of the list of children of the node. insertBefore( newChild, refChild)
Takes the node newChild and inserts it immediately before the refChild node you supply. replaceChild( newChild, oldChild)
Replaces the oldChild with the newChild, and oldChild is returned to the caller. removeChild( oldChild)
Removes the node oldChild from the list of children of the node this is called on.
IT-SC book 111 The brief descriptions do not replace the reference documentation for these methods; see Appendix D for more complete information.
4.5.3 Removing Nodes
Let's look at how to examine a tree, and how to remove specific nodes on the tree. Example 4-4 uses a few nested loops to dive three levels deep into an XML document created using the index.py script from Example 3-4. The design has its limitations, as it assumes you are only dealing with elements no more than three levels deep, but demonstrates the DOM methods we're interested in.
Example 4-4. domit.py
#!/usr/bin/env python import sys
from xml.dom.ext.reader.Sax2 import FromXmlStream from xml.dom.ext import PrettyPrint
# get DOM object doc = FromXmlStream(sys.stdin)
# remove unwanted nodes by traversing Node tree
for node1 in doc.childNodes:
for node2 in node1.childNodes:
node3 = node2.firstChild
while node3 is not None:
next = node3.nextSibling
name = node3.nodeName
if name in ("contents", "extension", "userID", "groupID"):
# remove unwanted nodes here via the parent
node2.removeChild(node3)
node3 = next
IT-SC book 112
PrettyPrint(doc)
After getting a document from standard input, a few nested for loops are executed to descend three levels deep into the tree and look for specific tag names. When running the script against the XML document we created with index.py, your file elements should look like this:
The whitespace around the removed elements remains in place as you can see by the gaps between elements; we did not look for adjacent text nodes, so they remain unaffected. This text was the result of a call to the PrettyPrint function at the end of the script. Of course, the element looks the same regardless of hierarchical position within the document. When writing DOM processing code, you should try to keep it independent from the structure of the document. Instead of using firstChild to get what you're after, consider enumerating the children and examining each one. This may cost some processing time, but it does give the document's structure more flexibility. As long as the target element appears beneath the parent node, the child will be found. When you use firstChild, you might be setting yourself up for trouble if someone gives you a document with a slightly different structure, such as a peer element coming before another in the document. You can write this type of operation using a recursive function, so that you can handle similar structures, regardless of position in the document. If you really don't care where within the subtree an element is found, you can use the getElementsByTagName method described earlier.
Another common requirement is to locate a node that you know must be a child of a particular node, but not require a specific ordering of the child nodes. A simple loop in a utility function handles this nicely: from xml.dom import Node
def findChildrenByTagName(parent, tagname):
IT-SC book 113 """Return a list of 'tagname' children of 'parent'."""
L = []
for child in parent.childNodes:
if (child.nodeType == Node.ELEMENT_NODE
and child.tagName == tagname):
L.append(child)
return L
An even simpler helper function that can come in handy is a function that finds the first child element with a particular tag name, or the first to have one of several tag names. These are all minor variations of the function just presented.
4.5.4 Changing a Document's Structure
In addition to doing replacements and additions, you can also restructure a document entirely using the DOM.
In Example 4-5, we take the nested loops from the last section, and replace them with a traveling recursive function. The script can also work with XML output from the index.py script we worked with earlier in this chapter. In this version however, the file element's size child is used as a replacement for itself. This process leaves the document filled with directory and size elements only.
Example 4-5 shows domit2.py using a recursive function.
Example 4-5. domit2.py
#!/usr/bin/env python
from xml.dom.ext.reader.Sax2 import FromXmlStream from xml.dom.ext import PrettyPrint
import sys
def makeSize(nodeList):
for subnode in nodeList:
if subnode.nodeType == subnode.ELEMENT_NODE:
IT-SC book 114 if subnode.nodeName == "size":
subnode.parentNode.parentNode.replaceChild(
subnode, subnode.parentNode)
else:
makeSize(subnode.childNodes)
# get DOM object doc = FromXmlStream(sys.stdin)
# call func makeSize(doc.childNodes)
# display altered document
PrettyPrint(doc)
You can run the script from the command line:
$> python domit2.py < wd.xml
The file wd.xml is an XML file created with the index.py script—you can use any file you like, as long as has the same structure as the files created by index.py. The output should be something like this:
IT-SC book 115
4.6 Building a Web Application
Now you can use your new knowledge of the DOM to create a simple web application. Let's build one that allows for the posting and viewing of articles. The articles are submitted and viewed via a web browser, but stored by the web server as XML, which allows the articles to be leveraged into different information systems that process XML. HTML articles, on the other hand, are unusable outside of a web browser.
4.6.1 Preparing the Web Server
In order to run the examples in this chapter, you must have a web server available that lets you execute CGI scripts. These examples were designed on Apache, so the CGI scripts contain a sh- bang line that specified the path to the Python executable (the #!/usr/bin/python expression at the top of the file) so that Apache can run them just like any other CGI script. (Understanding the term "sh-bang" requires a little bit of knowledge of Unix history. The traditional command- line environment for Unix was originally implemented using the sh program. The exclamation point was named the "bang" character because it was always used after words such as "bang" and "pow" in comic books and cartoons. Since the lines at the top of scripts that started with #! were interpreted by the sh program, they came to be known as sh-bang lines.)
4.6.1.1 Ensuring the script's execution
You must enable the execution of your Python scripts on your web server. On Apache, this means enabling CGI within the web directory, ensuring that the actual CGI scripts contain the pointer to the Python interpreter so they run correctly, and setting the "execute" permission on the script. This last item can be accomplished using the chmod program:
IT-SC book 116 $> chmod +x start.cgi
On other web servers and on Windows, you need to assign a handler to your CGI scripts so that they are executed by the Python interpreter. This may require that you name your scripts with a .py extension, as opposed to a .cgi extension, if .cgi is already assigned to another handler.
4.6.1.2 Enabling write permission
Beyond just being able to execute scripts within a web directory, the web user must also have write access to the directory for the examples to work. The examples are meant to illustrate the manipulation of XML and the ability to repurpose accessible XML into different applications.
To avoid dependency on a database in this chapter, and to provide easy access to the XML, these examples use the filesystem directly for storage. Articles are stored to disk as .xml files.
For Apache, you must give the user nobody write access to the specific web directory. If you are serving pages out of /home/httpd/myXMLApplication, you need to set up something like the following:
$> mkdir /home/httpd/myXMLApplication
$> chown nobody /home/httpd/myXMLApplication
$> chmod 755 /home/httpd/myXMLApplication
This gives the user nobody (the user ID that Apache runs under) write access to the directory. There are many other ways to securely set this up; this is simply one option. In general, for production web applications, it's a good idea not to give write access to web users.
4.6.2 The Web Application Structure
The web application is driven mainly by one script, start.cgi. The script does most of the processing, serves the content templates, and invokes the objects capable of storing and retrieving your XML articles. The primary components consist of the article object, the storage object, the article manager, the SAX-based article handler, and the start.cgi script that manages the whole process. Figure 4-2 shows a diagram of the major components.
Figure 4-2. The site architecture
IT-SC book 117 In the next few sections, we examine the code and operation of the CGI components in detail.
4.6.2.1 The Article class
The Article class represents an article as XML information. It's a thin class with methods only for creating an article from existing XML, or for retrieving the XML that makes up the article as a string. In addition, it has modifiable attributes that allow you to manipulate the content of the article:
def __init__(self):
"""Set initial data attributes."""
self.reset( )
def reset(self):
self.title = ""
self.size = 0
self.time = "" # pretty-printing time string
self.author = ""
self.contributor = ""
self.contents = ""
The attributes can be modified during the life of an article to keep you from having to create XML in your program. For example:
>>> from article import Article
>>> art = Article( )
>>> art.title = "FirstPost"
>>> art.contents = "This is the first article."
>>> print art.getXML( )
This is the first article.
IT-SC book 118
The getXML method call has the logic to recreate the XML when necessary. You can create articles with a well-formed string of XML, or by loading a string of XML from a disk file. The getXML method exists as a means for you to pull the XML back out of the object. Note the use of the escape function, which we imported from the xml.sax.saxutils module; this ensures that characters that are syntactically significant to XML are properly encoded in the result.
def getXML(self):
"""Returns XML after re-assembling from data
members that may have changed."""
attr = ''
if self.title:
attr = ' title="%s"' % escape(self.title)
s = '\n
if self.author:
s = '%s
if self.contributor:
s = '%s
if self.contents:
s = ('%s
% (s, escape(self.contents)))
return s + "\n"
The fromXML method of the article class populates the current XML article object with the values from the supplied string. This method uses the convenience function parseString, from xml.dom.minidom, to load the XML data into a document object, and then uses the content retrieval methods of the DOM to collect the required information:
def fromXML(self, data):
"""Initialize using an XML document passed as a string."""
self.reset()
dom = xml.dom.minidom.parseString(data)
IT-SC book 119 self.title = get_attribute(dom, "article", "title")
self.size = int(get_attribute(dom, "size", "bytes") or 0)
self.time = get_attribute(dom, "time", "stime")
self.author = get_attribute(dom, "author", "name")
self.contributor = get_attribute(dom, "contributor", "name")
nodelist = dom.getElementsByTagName("contents")
if nodelist:
assert len(nodelist) == 1
contents = nodelist[0]
contents.normalize()
if contents.childNodes:
self.contents = contents.firstChild.data.strip()
This method uses a convenience function defined elsewhere in the module. The function get_attribute looks into the document for an attribute and returns the value it finds; if the attribute it is looking for does not exist (or the element it expects to find it on does not exist), it returns an empty string instead. If it finds more than one element that matches the requested element type, it complains loudly using the assert statement. (For a real application, you would not use assert in this way, but this is sufficient for our examples since we're mainly interested in the XML aspect.)
When working with the web site logic, most manipulation on article objects occurs by either using the Storage class to load an article from disk, or by parsing a form submission to create an article for a user and then using the Storage class to save the XML file to disk. Example 4-6 shows the complete listing of the Article class.
Example 4-6. Article class from article.py import xml.dom.minidom from xml.sax.saxutils import escape
class Article:
"""Represents a block of text and metadata created from XML."""
def __init__(self):
IT-SC book 120 """Set initial data properties."""
self.reset( )
def reset(self):
"""Re-initialize data properties."""
self.title = ""
self.size = 0
self.time = "" # pretty-printing time string
self.author = ""
self.contributor = ""
self.contents = ""
def getXML(self):
"""Returns XML after re-assembling from data
members that may have changed."""
attr = ''
if self.title:
attr = ' title="%s"' % escape(self.title)
s = '\n
if self.author:
s = ('\n'
'
if self.author:
s = '%s
if self.contributor:
s = '%s
IT-SC book 121 if self.contents:
s = ('%s
% (s, escape(self.contents)))
return s + "\n"
def fromXML(self, data):
"""Initialize using an XML document passed as a string."""
self.reset( )
dom = xml.dom.minidom.parseString(data)
self.title = get_attribute(dom, "article", "title")
self.size = int(get_attribute(dom, "size", "bytes") or 0)
self.time = get_attribute(dom, "time", "stime")
self.author = get_attribute(dom, "author", "name")
self.contributor = get_attribute(dom, "contributor", "name")
nodelist = dom.getElementsByTagName("contents")
if nodelist:
assert len(nodelist) == 1
contents = nodelist[0]
contents.normalize( )
if contents.childNodes:
self.contents = contents.firstChild.data.strip( )
# Helper function:
def get_attribute(dom, tagname, attrname):
"""Return the value of a solitary element & attribute,
if available."""
nodelist = dom.getElementsByTagName(tagname)
IT-SC book 122 if nodelist:
assert len(nodelist) == 1
node = nodelist[0]
return node.getAttribute(attrname).strip( )
else:
return ""
4.6.2.2 The Storage class
The Storage class is used to place an article on disk as an XML file, and to create article objects from XML files that are already on disk:
>>> from article import Article
>>> from storage import Storage
>>> a = Article( )
>>> a.title = "FirstPost"
>>> a.contents = "This is the FirstPost."
>>> a.author = "Fred L. Drake, Jr."
>>> s = Storage( )
>>> s.save(a)
>>>
>>> b = s.load("FirstPost.xml")
>>> print b.getXML( )
This is the FirstPost.
IT-SC book 123 Here, you create an article from scratch as a, store it to disk using the Storage object, and then reincarnate the article as b using Storage's load method. Note that the load method takes the actual filename that is a concatenation of the article.title and the .xml extension.
The Storage.save method takes an article instance as the only parameter and saves the article to disk as an XML file using the form article.title.xml:
sFilename = article.title + ".xml"
fd = open(sFilename, "w")
# write file to disk with data from getXML( ) call
fd.write(article.getXML( ))
fd.close( )
The getXML method is used to retrieve an XML string containing an XML version of the article; the string is then saved to the disk file. The Storage.load method takes an XML file from disk, reads in the data from the file, and then creates an article using the fromXML method of the Article class:
fd = open(sName, "r")
sxml = fd.read( )
fd.close( )
# create an article instance
a = Article( )
a.fromXML(sxml)
# return article object to caller
return a
The return result is an Article instance. Example 4-7 shows storage.py in its entirety.
Example 4-7. storage.py
# storage.py from article import Article
IT-SC book 124 class Storage:
"""Stores and retrieves article objects as XML files
-- should be easy to migrate to a database."""
def save(self, article):
"""Save as
sFilename = article.title + ".xml"
fd = open(sFilename, "w")
# write file to disk with data from getXML( ) call
fd.write(article.getXML( ))
fd.close( )
def load(self, sName):
"""Name must be filename.xml--Returns an article object."""
fd = open(sName, "r")
sxml = fd.read( )
# create an article instance
a = Article( )
# use fromXML to create an article object
# from the file's XML
a.fromXML(sxml)
fd.close( )
# return article object to caller
return a
IT-SC book 125 4.6.3 Implementing Site Logic
The Article and Storage classes are not web-oriented. They could be used in any type of application, as the articles are represented in XML, and the Storage class just handles their I/O to disk. Conceptually at least, you could use these classes anywhere to create an XML-based information store.
On the other hand, you could write a single CGI script that has all of the logic to store articles to disk and read them, as well as parse the XML, but then your articles and their utility would be trapped within the CGI script. By breaking core functionality off into discrete components, you're free to use the Article and Storage classes from any type of application you envision.
In order to manage web interaction with the article classes, we will create one additional class (ArticleManager) and one additional script (start.cgi). The ArticleManager class builds a web interface for article manipulation. It has the ability to display articles as HTML, to accept posted articles from a web form, and to handle user interaction with the site. The start.cgi script handles I/O from the web server and drives the ArticleManager.
4.6.3.1 The ArticleManager class
The ArticleManager class contains four methods for dealing with articles. The manager acts as a liaison between the article objects and the actual CGI script that interfaces with the web server (and, indirectly the user's browser).
The viewAll method picks all of the XML articles off the disk and creates a section of HTML hyperlinks linking to the articles. This method is called by the CGI script to create a page showing all of the article titles as links: def viewAll(self):
"""Displays all XML files in the current
working directory."""
print "
View All
"
# grab list of files in current directory
fl = os.listdir(".")
for xmlFile in fl:
# weed out XML files
tname, ext = os.path.splitext(xmlFile)
if ext == ".xml":
# create HTML link surrounding article name
IT-SC book 126 print '
%s
'
% (quote(xmlFile), tname)
The method is not terribly elegant. It simply reads the contents of the current directory, picks out the XML files, and strips the .xml extension off the name before displaying it as a link. The link connects back again to the same page (start.cgi), but this time with query string parameters that instruct start.cgi to invoke the viewOne method to view the content of a single article. The quote function imported from urllib is used to escape special characters in the filename that may cause problems for the browser. URL construction and quoting is discussed in more detail in Chapter 8.
The viewOne method uses the storage object to reanimate an article stored on disk. Once the article instance is created, its data members are mined (one by one), and wrapped with HTML for display in the browser: def viewOne(self, articleFile):
""" takes an article file name as a parameter and
creates and displays an article object for it.
"""
# create storage and article objects
store = Storage( )
art = store.load(articleFile)
# Write HTML to browser (standard output)
print "
Title: " + art.title + "
"
print "Author: " + art.author + "
"
print "Date: " + art.time + "
"
print "
| " + art.contents print " |
It's important to note here that the parameter handed to viewOne is a real filename, not just the title of the XML document.
The postArticle method is probably the simplest method discussed yet, as its job is simply to create HTML. The HTML represents a submittal form whereby users can write new articles and present them to the server for ultimate storage in XML. Since the HTML form does not change, this method can simply print the value of a constant that contains the form as a string.
IT-SC book 127 The postArticleData method is slightly more complicated. Its job is to extract key/value pairs from a submitted HTTP form, and create an XML article based on the obtained values. Once the XML is created, it must be stored to disk. It does this by creating an article object and setting the members to values retrieved from the form, then using the Storage class to save the article. def postArticleData(self, form):
"""Accepts actual posted form data, creates and
stores an article object."""
# populate an article with information from the form
art = Article()
art.title = form["title"].value
art.author = form["author"].value
art.contributor = form["contrib"].value
art.contents = form["contents"].value
# store the article
store = Storage()
store.save(art)
Example 4-8 shows ArticleManager.py in its entirety.
Example 4-8. ArticleManager.py
# ArticleManager.py import os from urllib import quote from article import Article from storage import Storage
class ArticleManager:
"""Manages articles for the web page.
IT-SC book 128 Responsible for creating, loading, saving, and displaying articles."""
def viewAll(self):
"""Displays all XML files in the current working directory."""
print "
View All
"
# grab list of files in current directory
fl = os.listdir(".")
for xmlFile in fl:
# weed out XML files
tname, ext = os.path.splitext(xmlFile)
if ext == ".xml":
# create HTML link surrounding article name
print '
%s
' \
% (quote(xmlFile), tname)
def viewOne(self, articleFile):
"""Takes an article file name as a parameter and
creates and displays an article object for it.
"""
# create storage and article objects
store = Storage( )
art = store.load(articleFile)
# Write HTML to browser (standard output)
print "
Title: " + art.title + "
"
print "Author: " + art.author + "
"
IT-SC book 129 print "Date: " + art.time + "
"
print "
| " + art.contents print " |
def postArticle(self):
"""Displays the article posting form."""
print POSTING_FORM
def postArticleData(self,form):
"""Accepts actual posted form data, creates and
stores an article object."""
# populate an article with information from the form
art = Article()
art.title = form["title"].value
art.author = form["author"].value
art.contributor = form["contrib"].value
art.contents = form["contents"].value
# store the article
store = Storage()
store.save(art)
POSTING_FORM = '''\
'''
4.6.4 Controlling the Application
The CGI script is the main program for the web application. It is also the only "page" that will ever be in the browser. When the user types start.cgi in the address bar, Apache runs the script on the server.
The script begins by importing the cgi and os modules: import cgi import os
The script then prints the content header, as well as the opening HTML. This HTML is the same regardless of the type of operation start.cgi is performing; therefore, it is defined as the constant HEADER (not shown) and printed for every request:
# content-type header print "Content-type: text/html" print print HEADER
After the common portion of the result page is printed, the query string is checked for the cmd parameter, which specifies what actions start.cgi should perform. The hyperlinks produced and sent to the browser by start.cgi are all fitted with this same parameter indicating a specific instruction such as view or post. The query string is checked using the cgi module. It is inspected to see if it contains the cmd parameter. If so, processing continues; if not, the user is presented with an error message. query = cgi.FieldStorage( ) if query.has_key("cmd"):
cmd = query["cmd"][0].value
# instantiate an ArticleManager
am = ArticleManager( )
IT-SC book 131 The ArticleManager is instantiated as am, and command processing continues by checking cmd for its four possible values. For viewing article titles, the command sequence va is used:
# Command: viewAll - list all articles if cmd == "va":
am.viewAll( )
For viewing a specific article, the command sequence v1a is used:
# Command: viewOne - view one article if cmd == "v1a":
aname = query["af"].value
am.viewOne(aname)
For posting articles, a form is displayed. The CGI script looks for the pa sequence:
# Command: postArticle - view the post-article page
if cmd == "pa":
am.postArticle( )
When the user submits the article form, the data is posted to the web server. The CGI script looks for the command sequence pd to indicate that the article data is posted. It then passes the CGI form to the ArticleManager's postArticleData method:
# Command: postData - take an actual article post if cmd == "pd":
print "
Thank you for your post!
"am.postArticleData(query)
If cmd is not present in the query string, or if cmd has a value that is not one of the four, an error message is presented as the else clause to the first if statement: else:
# Invalid selection
print "
Your selection was not recognized
"The HTML is then closed by a final print statement:
# close the HTML print ""
IT-SC book 132 The complete listing of start.cgi is shown in Example 4-9.
Example 4-9. start.cgi
#!/usr/local/bin/python
#
# start.cgi - a Python CGI script
import cgi import os
from ArticleManager import ArticleManager
HEADER = """\
XML Articles |
View All | |
"""
#
# MAIN
IT-SC book 133 #
# content-type header print "Content-type: text/html" print print HEADER
# retrieve query string query = cgi.FieldStorage( ) if query.has_key("cmd"):
cmd = query["cmd"].value
# instantiate an ArticleManager
am = ArticleManager( )
# do something for each command
# Command: viewAll - list all articles
if cmd == "va":
am.viewAll( )
# Command: viewOne - view one article
if cmd == "v1a":
aname = query["af"].value
am.viewOne(aname)
# Command: postArticle - view the post-article page
if cmd == "pa":
IT-SC book 134 am.postArticle( )
# Command: postData - take an actual article post
if cmd == "pd":
print "
Thank you for your post!
"am.postArticleData(query)
else:
# Invalid selection
print "
Your selection was not recognized.
"# close the HTML print ""
Take note of the initial #!/usr/local/bin/python expression. As this is a CGI script, the operating system needs a hint on how to run it. If it is compiled C code, it could be executed by the web server; however, if it is a script, it likely needs to be handed off to the services of a script interpreter. Such is the case with Python. Note that we did not use the sh-bang line #!/usr/bin/env python; that could open a security hole when used with CGI scripts. See the documentation of Python's cgi module for more information about CGI security issues and how to address them properly when using Python.
4.7 Going Beyond SAX and DOM
In this chapter, we discussed the DOM and how it differs from SAX. In the next chapter, we explore another method of extracting interesting portions of an XML document using a basic traversal language called XPath. Once you've learned a little about XPath, we move on in Chapter 6 to a transformation technology called XSLT.
IT-SC book 135 Chapter 5. Querying XML with XPath
The XML Path Language (XPath) is a language that allows you to easily perform searches against XML documents using a path-like string. XPath searches return individual nodes or collections of nodes based on expressions. XPath does not use XML syntax. In fact, it is not a procedural programming language in the normal sense; there is no concept of control flow. XPath expressions are usually single strings. XPath does, however, support some functional manipulation (usually found in programming languages) but one doesn't write XPath scripts or programs. Instead, one writes XPath expressions that are evaluated against XML documents and result in node lists being returned. In this chapter, we discuss the origin of XPath, as well as its syntax, capabilities, and how it is used from within Python.
5.1 XPath at a Glance
XPath 1.0 is a W3C recommendation available for your perusal from the W3C web site (http://www.w3.org/TR/xpath). XPath allows for the retrieval of portions of an XML document via XPath expressions. The specification defines a concrete syntax for expressions and offers a well-defined meaning for the expressions when interpreted. When an XPath expression is processed with a DOM, the nodes that match the expression are returned to the caller. XPath expressions target a specific node, or groups of nodes, within an XML document. The result is one of four types:
• A collection of nodes • A Boolean value • A floating-point number • A string
In XPath, the term context refers to the location in the document where the XPath expression is being applied. You may start from the document element (the root element) or from any descendent element. XPath may inform you of the current context node (representing the current location in the document). A pair of integers may represent context position and context size. There may also be variable bindings in the context, available functions, and a namespace relevant to the current position.
5.2 Where Is XPath Used?
XPath is not stored within a particular type of document. Instead, XPath expressions are used primarily in XSLT (a transformation language used with XML), but can be used elsewhere as well. (XSLT is covered in more detail in Chapter 6.) In the case of APIs such as 4XPath, expressions can be used against a DOM to return results programmatically in your Python programs. Microsoft's MSXML3.0 (covered in Appendix E) processes XPath expressions as well.
5.3 Location Paths
The most commonly used type of XPath expression is the location path. A location path can be thought of as similar to a path for a file on a disk, but on steroids. Where a path for a filesystem contains only names of directories and a file, an XPath
IT-SC book 136 location path can specify much more. At each step along the path, it can perform selection based on complex tests of the nodes in a document, and the result may be several nodes. The tests, or predicates, for each step of the path can match based on element name, attribute presence or value, or textual content.
The full syntax of location paths is complex, but the specification is considerate enough to define abbreviated forms for the most commonly used tests; these are called abbreviated location paths. All of the location paths we describe in this chapter use the abbreviated syntax; for more information on the full syntax and selection capabilities of XPath, please refer to the specification.
Location paths are used within XSLT elements, but may also be used programmatically with an XPath API to return node sets from an XML document at runtime. The latter technique will come into greater focus as you read this chapter; the former is covered in Chapter 6.
5.3.1 An Example Document
Let's start with an example document that represents data records. The records are all fairly similar, but of course the field values are different in each one. This is typical of the type of documents you might mine with XPath. In Example 5-1, we apply XPath expressions against an XML document representing starships from some popular science-fiction television series.
Example 5-1. ships.xml
name="United Federation of Planets"> IT-SC book 137
5.3.2 A Path Hosting Script
The ships.xml file provides a good stretch of XML data to write paths against. Now you can write a small program to apply path expressions to the document, and report on the nodes that are returned. In Example 5-2, we create a small script, xp.py, which invokes the xml.xpath.Evaluate function provided with 4Suite and more recent versions of PyXML.
Example 5-2. xp.py
""" xp.py (requires xml doc on stdin)
""" import sys
from xml.dom.ext.reader import PyExpat from xml.xpath import Evaluate
IT-SC book 138
path0 = "ship/captain" # all captain elements
reader = PyExpat.Reader( ) dom = reader.fromStream(sys.stdin)
captain_elements = Evaluate(path0, dom.documentElement) for element in captain_elements:
print "Element: ", element
To run this program, you need to supply the previously created ships.xml from Example 5-1 as input:
$ python xp.py < ships.xml
In Example 5-2, the path ship/captain is used to extract all captain elements from the ships.xml document. The result is a node list containing the following:
Of course, this is not a complete or standalone document, but rather a node list. These nodes are processed by the remaining code in the program: captain_elements = Evaluate(path0, dom.documentElement) for element in captain_elements:
print "Element: ", element
The path ship/captain is a relative location path, as it does not specify an exact location from the root of the document to the element, as does /shiptypes/ship/captain. The ship/captain expression returns captain elements that are children of a ship element, relative to the document node passed to Evaluate.
5.3.3 Getting Character Data
IT-SC book 139 You will often want to target text beneath an element. For example, you may want to search just for the captain's name, rather than the element node. You could append the XPath text function to your expression: path1 = "ship/captain/text( )"
This addition to the path expression selects all text nodes beneath the captain element. If you replace the original production lines with the following code: captainnodes = Evaluate(path1, dom.documentElement) for captainnode in captainnodes:
print "Starfleet Captain: ", captainnode.nodeValue you see the following result:
$ python xp.py < ships.xml
Starfleet Captain: Jean-Luc Picard
Starfleet Captain: Kathryn Janeway
Starfleet Captain: Jean-Luc Picard
Starfleet Captain: James T. Kirk
Starfleet Captain: Benjamin L. Sisko
5.3.4 Specifying an Index
Often, when working with data, you become interested in the ordinal positions of elements within columns, rows, or arrays. XML is no different in this regard. XPath provides indexed elements with syntax similar to array indexes, but it is important to know that XPath indexes are one-based, while Python sequence indexes are zero- based. To target an element using an index, use brackets next to the element name: path2 = "ship[2]/captain/text( )"
In this case, ship[2] indicates that the second ship element for each parent of any ship element should have the text nodes beneath its captain element selected. To see the output, change the processing code: capnode = Evaluate(path2, dom.documentElement) print "Captain of ship[2] is: ", capnode[0].nodeValue
Using path2, the output is:
$ python xp.py < ships.xml
Captain of ship[2] is: Kathryn Janeway
IT-SC book 140 It is important not to allow the visual similarity between ship[2] and Python sequence indexing to confuse you; they are very different. The notation is actually shorthand for ship[position( )=2], which indicates that the second ship child element of some other element will match. Consider the following XML fragment:
The XPath expression ship[2] matches only the ship element with an id attribute of id5. This is not a trick, but it is an excellent reason to keep a copy of the XPath specification close by.
5.3.5 Testing Descendent Nodes
You may also want to query the text content beneath an element name. Say you have a structure of book chapters, each containing headings and paragraphs. You may want to search for text that appears underneath a certain heading. XPath provides a convenient way for you to check the character data of a text node that is the child of an element. If you are searching for a
This expression selects ship elements that have a child class element with child character data of Intrepid. You can further explore the returned node list with a processing code: shipnodes = Evaluate(path3, dom.documentElement) for shipnode in shipnodes:
shipname = shipnode.getAttribute("name")
captain = Evaluate("captain/text( )", shipnode)
print "------Intrepid Class Ship ------"
IT-SC book 141 print "Name: ", shipname
print "Captain: ", captain[0].nodeValue
In this code, we select all ship nodes that have a child class element indicating that they are Intrepid class ships. We can then reprocess this node to further select ship names and captains to generate the following output:
$ python xp.py < ships.xml
------Intrepid Class Ship ------
Name: USS Voyager
Captain: Kathryn Janeway
Instead of just checking that a descendent element contains necessary information as in path3, you can continue building the path expression to grab something specific beneath the matching element: path4 = 'ship[class="Constitution"]/@name'
In this path, you drill down further. First, a ship element is selected only if its child class element contains the character data Constitution. This path is further extended when we select the name attribute of the ship element that contains the specific child character data (the @ symbol is used to indicate that we're interested in an attribute rather than a child element). Again, we change the processing code a little to use the new node list: ship = Evaluate(path4, dom.documentElement) print "Name of Constitution Class Ship: ", ship[0].nodeValue
The output follows:
$ python xp.py < ships.xml
Name of Constitution Class Ship: USS Enterprise
5.3.6 Testing Attributes
Of course, evaluating XML attributes and their contents involves a slightly different process than evaluating element names and text node character data. In XPath, the @ character is used to indicate an attribute. Brackets are also used to surround the node when it is being tested against character data. In order to test the character contents of an attribute, use a path such as the following: path5 = 'ship[@name="USS Enterprise"]'
This expression selects all ship elements that have a name attribute containing the word Enterprise. In your ships.xml file, there are three starships named Enterprise, each with slightly different registry codes. You can mine the node list for more information:
IT-SC book 142 ships = Evaluate(path5, dom.documentElement) for shipnode in ships:
registry = Evaluate("registry-code/text( )", shipnode)
captain = Evaluate("captain/text( )", shipnode)
print "Found Enterprise with registry: ", registry[0].nodeValue
print "Captain: ", captain[0].nodeValue
These subsequent expressions are relative paths that select captain and registry- code text from the current element with each hop through the node list. This time using the preceding code, the output appears as:
$ python xp.py < ships.xml
Found Enterprise with registry: NCC-1701-E
Captain: Jean-Luc Picard
Found Enterprise with registry: NCC-1701-D
Captain: Jean-Luc Picard
Found Enterprise with registry: NCC-1701
Captain: James T. Kirk
5.3.7 Selecting Elements
As with any ordered data set, you are usually interested in pulling one specific type of information out from the entire document. You may only be interested in the names of employees in a human resources database. Or you may have heavily nested data that you want to make sure you pull out with each occurrence of a given data type, regardless of its position in the document. With XPath, you can use the path expression // to indicate that all matching elements beneath the root should be selected: path6 = "/shiptypes//captain"
This expression selects all captain elements beneath the route, regardless of where they appear. Since you are working with elements, obtaining character data requires some of the work shown earlier, or a traversal of the node structure: captains = Evaluate(path6, dom.documentElement) for captain in captains:
print "Captain: ", captain.firstChild.nodeValue
Running path6 generates the following output:
IT-SC book 143 $ python xp.py < ships.xml
Captain: Jean-Luc Picard
Captain: Kathryn Janeway
Captain: Jean-Luc Picard
Captain: James T. Kirk
Captain: Benjamin L. Sisko
5.3.8 Additional Operators
If you are familiar with filesystem paths on Windows or Unix, you may have seen the . and .. operators. The . operator indicates the current directory (or current element in XPath) while .. refers to the parent directory (or parent element in XPath). Using ships.xml, shown in Example 5-1, we can search for a specific ship's name and then reference the parent element to see which organization the ship belongs to. path7 = "ship[@name='USS Voyager']/../@name"
This expression searches for a ship element that has a name attribute of "USS Voyager." The path then continues to select the name attribute of this ship element's parent. In ships.xml, this is the name attribute of the shiptypes element. To generate output, change your processing code in xp.py: org = Evaluate(path7, dom.documentElement) print "USS Voyager is owned by", org[0].nodeValue
This time xp.py generates output attributing the Voyager to the Federation of Planets:
$ python xp.py < ships.xml
USS Voyager is owned by United Federation of Planets
5.4 XPath Arithmetic Operators
In addition to selecting elements by location paths, XPath also provides capability for data manipulation. The numerical parts of an XML document can be added, divided, subtracted, and multiplied. Likewise, strings can be compared for equality.
XPath provides arithmetic operators for use within XPath expressions. This capability comes in very handy in XSL transformations that involve totaling an item list or applying discounts to product prices for display in HTML. The operators available in XPath are +, -, *, div, and mod (addition, subtraction, multiplication, division, and modulus, respectively.) There are also functions such as sum that allow you to total sets of numbers and perform other tasks. We cover functions in the next section.
IT-SC book 144 Imagine that you have an XML file containing a list of products, and you want to display these products in another application (such as your web site) but need to apply a 20% discount to all retail prices. You can use the XPath arithmetic operators to solve this problem. Let's turn to the source XML document (products.xml) shown in Example 5-3.
Example 5-3. products.xml
To apply a blanket 20% discount to all products, you can use XPath from within an XSLT document. The XSLT shown in Example 5-4 (products.xsl) does the trick.
Example 5-4. products.xsl
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
Item:
Orig. Price:
IT-SC book 145
The XPath numerical expressions are in the xsl:value-of elements. The discount is achieved by multiplying the value of the price attribute by 0.8. You can run the transformation using the 4xslt tool illustrated in the previous chapter:
$ 4xslt.bat products.xml products.xsl
Item: bowl
Orig. Price: 19.95, Our Price: 15.96
Item: spatula
Orig. Price: 4.95, Our Price: 3.96
Item: power mixer
Orig. Price: 149.95, Our Price: 119.96
Item: chef hat
Orig. Price: 39.95, Our Price: 31.96
The div and mod operators work as the others do. For example, @price div 2 divides all prices designated by 2.
5.5 XPath Functions
IT-SC book 146 XPath provides numerous functions for working with numbers and strings, and allows you to complete transformations and mine XML data without having to constantly bridge other APIs or technologies to do simple string and arithmetic operations. Adding, simple division, multiplication, and string searching are available as built-in functions of XPath.
5.5.1 Working with Numbers
Several XPath functions are available to you. In Example 5-4, multiplication is used to apply a 20% discount to products. If you need to total a list of products, you can use the sum function, working with the same products data again:
This time, in Example 5-5, you can use a single XPath expression to generate a total. The expression sum(//@price) returns the sum of the values of all price elements in the products document. Now go back and modify the stylesheet you created to discount the products, but this time add in an xsl:value-of element to generate a total.
Example 5-5. products.xsl
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
Your Total:
IT-SC book 147
Figure 5-1 shows the result of the transformation (the HTML) in a browser.
Figure 5-1. Using the sum( ) XPath function
In addition to sum, several other functions exist for working with numbers. The floor function returns the largest integer that is not greater than the argument. In other words, floor(3.4) returns 3. The ceiling function, floor's counterpart, returns the smallest integer that is greater than the argument, e.g., ceiling(3.4) returns 4.
IT-SC book 148 The round function does exactly what you think it should: round your decimal-ridden number to its closest integer. For example, round(3.4) returns 3, while round(3.8) returns 4.
5.5.2 Working with Strings
In addition to functions for numbers, XPath supports functions for manipulating text. Most of these are valuable when doing conditional testing. Earlier you checked character data of child attributes with syntax such as: ship[class="Intrepid"]
This expression returns any ship element with a class element beneath it containing the character data Intrepid. This is a fine approach for exact comparisons, but sometimes you'll want finer control.
For example, the starts-with function takes two arguments. The first argument is what you're looking for: the letters the string may start with. The second argument is the node to evaluate. The function returns true or false. For example, to get a true or false return (in XSL) on whether a ship element has a registry code that starts with NCC, you can try the following expression:
This expression returns true for every ship in the ships.xml file. This type of Boolean return may be of most benefit in XSLT, where you can use its if-then-else language features for conditional processing. A variation on this theme is the contains function, which returns true if the second argument contains the first argument.
If you know you have the string you want and are looking to slice and dice it, the substring and string-length functions can help you out. The substring function takes up to three arguments. The first argument is the string to manipulate; the second argument is the starting index within the string; the third argument is the ending index. If the third argument is omitted, it's assumed to be the end of the string. The string-length function is straightforward, and returns the total length of the string as a number.
The translate function takes a string parameter, as well as a list of characters to replace and a list of corresponding replacement characters. Each character in the second argument is replaced by the corresponding character in the same position in the third argument. For example, the expression translate("Wee Willy Winky", "eily", "oaps") returns the string Woo Wapps Wanks. The concat function returns the concatenation of its two arguments.
5.5.3 Working with Nodes
Some functions in XPath are designed to work with elements and element traversal itself. These functions supply information related to XPath's current position, and other positional type of information such as first matching element and last matching element. Node functions are fairly straightforward.
IT-SC book 149 The position function returns a number equal to the context position from the expression evaluation context. For example, to create a numbered list for the ships of ships.xml, you could use the position function as shown in the following stylesheet:
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
This code generates the following HTML output:
1.
USS Enterprise
2.
USS Voyager
3.
IT-SC book 150 USS Enterprise
4.
USS Enterprise
5.
USS Sao Paulo
The count function returns the number of nodes in the node set matching the argument. In other words, count(//@name) returns the total number of name attributes within a document. The last function returns a number equal to the context size (the number of nodes) in the current expression.
The id function returns a node by specific id. If you create an element
5.6 Compiling XPath Expressions
In this chapter, we use the Evaluate function from the 4XPath API to apply XPath expressions against node sets. For programmatic use of XPath within Python, the 4XPath API is readily available and offers considerable power.
Most of the XPath API is geared towards supporting XPath expressions, as XPath is a standard. But for the programmer embedding XPath processing functionality into their applications, there is some optimization found in 4XPath.
The Compile and Context functions aid the developer to create compiled XPath expressions for repeated use against multiple documents. For example, if you are accepting large numbers of XML documents from customers or suppliers, you may want to apply an XPath expression to each one (as it arrives) to figure out what to do with it, or where to route it within your organization. Having your XPath expression readily compiled and applied against each unique document adds speed to your application, as you've done away with the need to parse the XPath expression.
The Compile function returns an expression object that supports an evaluate method similar to the Evaluate function used thus far in this chapter. However, the method expects a Context object, not a node. The task of compiling an expression, and then using the compiled version, is fairly simple: expression = Compile("ship/@name") context = Context(dom.documentElement)
IT-SC book 151 nodes = expression.evaluate(context)
The first step is to generate an expression; the second step is to generate context for the document or node set you're working with. You can run the expression by calling the evaluate method of the compiled expression object, as demonstrated in Example 5-6 (which makes use of the ships.xml file).
Example 5-6. xp.py
#!/usr/local/bin/python
import sys
from xml.dom.ext.reader import PyExpat from xml.xpath import Compile from xml.xpath.Context import Context
reader = PyExpat.Reader( ) dom = reader.fromStream(sys.stdin)
expression = Compile("ship/@name") context = Context(dom.documentElement) nodes = expression.evaluate(context)
print "Nodes: ", nodes
When executed from the command line with ships.xml as input, the program generates the following output:
$ python compx.py < ships.xml
Nodes: [
IT-SC book 152
Your Python and XML toolkit is almost complete; we look at one more core technology in the next chapter. After that, we delve into topics that deal with actually integrating XML with your existing systems and building new systems using Python and XML.
IT-SC book 153 Chapter 6. Transforming XML with XSLT
We've covered using SAX to capture XML parsing events and output corresponding HTML for display in a web browser. XML's power lies in its ability to represent data for data's sake. XML is not concerned with displays such as web pages, handheld devices, PostScript files, etc. Instead, XML is concerned only with the structure of your information. For this reason, we frequently parse XML and convert it to another format for viewing, such as HTML.
In this chapter, we discuss Extensible Stylesheet Language Transformations (XSLT). One of the simplest things that XSLT does is transform your XML documents into HTML documents for consumption by browsers. We go over how to construct an XSLT stylesheet that performs the same transformation for you that SAX did earlier, but with considerably less effort. Keep in mind, however, that XSLT is more powerful than mere HTML production as it transforms one XML document written for a specific DTD or dialect into another dialect. These XML to XML transformations can be very powerful when exchanging business documents between Internet domains that use different dialects. Dialects and validation are covered in Chapter 7.
6.1 The XSLT Specification
The XSLT specification is available from the World Wide Web Consortium web site at http://www.w3.org/TR/xslt.html. Reading the specification is a perfect cure for insomnia, so to keep you awake, I summarize key parts of XSLT here as they relate to Python.
In any working XSLT setup, three distinct files exist and at least one piece of software is utilized. The first is the source XML file, which is your original document. The next is the XSL stylesheet, which represents the rules of the transformation and is itself an XML-compliant document. The third and final document is the result of the transformation. This is most likely either HTML or XML. The essential software used to create the transformation is the XSLT processor. This software loads the original XML document, applies the transformation rules, and spits out the result of the transformation. Figure 6-1 shows an example of this arrangement.
Figure 6-1. The XSLT transformation process
The XSLT language is an XML-based language. It is defined as a set of elements and attributes with carefully defined semantics. XSLT is very straightforward, as you'll discover.
IT-SC book 154 6.2 XSLT Processors
There are a variety of XSLT processors available on the market, both free and commercial. The power of XSLT is in the transformations that the language allows, but the actual work is completed by the processor. Depending on your environment, you may choose a processor based on speed or on accessibility from a particular platform such as Python. Alternatively, you may choose a processor that you can drive programmatically.
The XSLT processor's job is to take an XSL stylesheet and perform its transformation rules against an existing XML document to produce a new transformed document. The W3C states that XSLT is for transforming XML to XML, which is true, but it can be used to generate HTML or other formats as well. It is frequently used to transform XML to HTML or XHTML for viewing in a web browser.
XSLT is a language unto itself, and has nothing in particular to do with Python. As such, you can convert documents for use in your Python programs with any XSLT processor. However, if you are hoping to embed XSLT functionality within your Python programs, you need a processor accessible from Python either natively (such as 4XSLT) or by a bridge mechanism (such as using MSXML3.0 from Python, as covered in Appendix E).
For Python, the 4XSLT package is an open source XSLT processor that can be driven from the command line as well as embedded in your Python programs—it is primarily implemented in Python, but includes some modules written in C for improved performance. 4XSLT is available from Fourthought, Inc. as part of the 4Suite package (see http://www.4suite.org/).
Other XSLT processors exist for other languages and platforms, but can still batch process transformations for use in your Python applications. Microsoft's Internet Explorer has an XSLT processor embedded within it, and can transform an XML document into HTML in the client's browser (though versions prior to 6.0 are horribly noncompliant). SAXON is a collection of XML tools, including a Java-based XSLT processor capable of running in any Java virtual machine. Sablotron is a fast C++ XSLT processor. The W3C's XSLT site (http://www.w3.org/Style/XSL/) contains numerous links to XSLT processing software.
For the remainder of this chapter, we use the 4XSLT processor as it's completely Python-based, and its functionality is accessible at runtime from your Python applications.
6.3 Defining Stylesheets
If you are familiar with the Cascading Style Sheets (CSS) specification often used on the Web, you are probably aware that CSS stylesheets can be stored in a separate file, or embedded as a special element within an HTML document. Also, specific styling information can be attached to individual attributes within the document. In this section, we examine the corresponding approaches to using XSLT.
Each of the three ways of using CSS have an analogous technique using XSLT, but the XSLT stylesheets are substantially more powerful. While this discussion refers to some specific XSLT elements and shows several in the examples, it does not expect
IT-SC book 155 that you know anything about them. These elements are described in more detail later in this chapter; this section simply introduces you to the ways stylesheets can be written and how that relates to the documents being processed.
6.3.1 Simplified Stylesheets
Simplified stylesheets are more like using the STYLE attribute in HTML documents than anything else, but the similarity is minimal. This approach is somewhat less powerful than using embedded or standalone stylesheets; the xsl:stylesheet element is not allowed since the entire stylesheet is interpreted as the body of an xsl:template element. Many features of XSLT require using additional "top-level" elements (peers of the xsl:template element), so they are not allowed in this context. This kind of stylesheet is more difficult to use when the basic structure of the source document needs to be preserved, but is perfectly able to make queries about the structure and content of the source document. Simplified stylesheets are most often applied when the output documents are very regular and only need to extract very specific portions of the input document.
Since simplified stylesheets are also about the easiest to start with when learning XSLT, let's take a look at one. In the previous chapter, we use a list of spaceships from a group of well-known television shows to provide input data (see Example 5-1); we use that input here as well. Instead of using the DOM and XPath to retrieve a list of nodes, we use XSLT to create a list of spaceships sorted by their registry numbers, nicely presented as an HTML table. Example 6-1 shows the stylesheet. Notice the root element of the stylesheet document declares the namespace for XSLT and specifies the XSLT version that is being used; these are required for the use of simplified stylesheets.
Example 6-1. ships-template.html
xsl:version="1.0">
| Ship | Class | Registration | Captain |
|---|---|---|---|
The result of processing the ships.xml file from Example 5-1 with the stylesheet ships-template.html in Example 6-1 is given in ships.html, shown in Example 6-2. The transformation was performed using 4XSLT.
Example 6-2. ships.html
| Ship | Class | Registration | Captain |
|---|---|---|---|
| USS Enterprise | Constitution | NCC-1701 | James T. Kirk |
| USS Enterprise | Galaxy | NCC-1701-D | Jean-Luc Picard |
| USS Enterprise | Sovereign | NCC-1701-E | Jean-Luc Picard |
| USS Voyager | Intrepid | NCC-74656 | Kathryn Janeway |
| USS Sao Paulo | Defiant | NCC-75633 | Benjamin L. Sisko |
Note that the transformation added a meta element near the top of the generated HTML, and that the indentation and whitespace inside the replacement for the xsl:for-each element has been adjusted somewhat. Figure 6-2 shows what the resulting HTML document looks like in a web browser.
Figure 6-2. ships.html in a browser
6.3.2 Standalone Stylesheets
Stylesheets stored in separate files are perhaps the most commonly used form of stylesheets for both CSS and XSLT. The root element of the stylesheet must be an xsl:stylesheet or xsl:transform element. This is what we use for most of the examples in this book. Standalone stylesheets offer more power and flexibility than simplified stylesheets, and lend themselves to better modularization, allowing use of a powerful import mechanism as well as strong pattern matching abilities.
Let's look at the previous example expressed as a standalone stylesheet. We could use a trivial wrapper around the template document to create a stylesheet that is technically correct, but let's go ahead and change it to reflect a more typical way of structuring a stylesheet. This particular version no longer sorts the table of ships, but maintains their order from the original document. This is a common way of structuring a stylesheet for a document-oriented application. Our new stylesheet is shown in Example 6-3. Notice that the XSLT namespace is declared here as well,
IT-SC book 159 along with the version attribute, but we need not include the namespace prefix when the attribute is attached to an xsl:stylesheet element.
Example 6-3. ships.xsl
version="1.0">
| Ship | Class | Registration | Captain |
|---|
This version is structured as a set of templates that match particular constructs in the input document; the matched constructs are specified by the match attribute of the xsl:template elements. The XSLT constructs used in this stylesheet are explained in detail later in this chapter. Example 6-4 shows the result of transforming ships.xml (see Example 5-1) using ships.xsl (see Example 6-3).
Example 6-4. ships2.html
| Ship | Class | Registration | Captain |
|---|---|---|---|
| USS Enterprise | Sovereign | NCC-1701-E | Jean-Luc Picard |
| USS Voyager | Intrepid | NCC-74656 | Kathryn Janeway |
| USS Enterprise | Galaxy | NCC-1701-D | Jean-Luc Picard |
| USS Enterprise | Constitution | NCC-1701 | James T. Kirk |
| USS Sao Paulo | Defiant | NCC-75633 | Benjamin L. Sisko |
The only difference between this output and Example 6-2 is that the table is not sorted in this version.
6.3.3 Embedded Stylesheets
XSLT stylesheets can be embedded within other documents in much the same way that CSS stylesheets can be embedded in an HTML document. When embedding an XSLT stylesheet, it is typically embedded in the document to which it applies. The embedded element must be the xsl:stylesheet (or xsl:transform) element. This usage pattern is not commonly used since it doesn't allow the stylesheet to be re- used as easily with other documents, and few XSLT processors support embedded stylesheets. Given the lack of broad tool support for embedded stylesheets, we won't bother showing any examples.
6.4 Using XSLT from the Command Line
Before we learn how to embed XSLT transformations in Python programs, we need to concentrate on learning more about XSLT itself. As we're learning, it will generally be easier to run our transformations from the command line than from a Python script. Many of the processors provide a command-line tool for performing transformations. We use the 4XSLT package provided as part of 4Suite; if you choose to use a different tool, please consult its documentation to determine how to use it.
4XSLT includes a script that performs transformations from the command line. On Windows, the 4xslt.bat script is installed in the Scripts directory of your Python installation by the 4Suite installer. To make the script more easily usable, either add the Scripts directory to your PATH environment variable or copy the 4xslt.bat file to a directory that is already included in the PATH.
The basic operation of 4xslt simply requires two parameters: the XML document to transform, and the stylesheet to apply. This example was used to apply the stylesheet from Example 6-3 to produce the output shown in Example 6-4:
IT-SC book 163 C:\my-dir> 4xslt ships.xml ships.xsl > ships2.html
Output redirection is used to save the result of the transformation to a file.
4XSLT and the 4xslt script support the use of both simplified and standalone stylesheets. Embedded stylesheets are not supported.
6.5 XSLT Elements
Much of XSLT's functionality is exercised in the form of elements that perform functions and tasks. In fact, the whole language is XML-based and describing its features is already the subject of several books. This section presents some of the XSLT elements and fundamentals so you can begin using it in your daily work.
6.5.1 The Stylesheet Element
The xsl:stylesheet element is always the root element of standalone stylesheets, and is also used for embedded stylesheets. The stylesheet element contains some optional and mandatory attributes that provide more details about the stylesheet to the XSLT processor. The specification defines a second root element in the XSLT namespace, called xsl:transform. This element is identical to xsl:stylesheet in every way but name, and can be used in place of xsl:stylesheet with no change in meaning.
The id attribute is optional. However, an identifier would certainly come in handy if this stylesheet were part of a larger XML document (as would be the case for an embedded stylesheet). The XML specification states that any attribute of type ID (not necessarily named id, but of the data type ID declared in the DTD) must be unique within any XML document. Use of an ID attribute on a stylesheet is powerful if you are dynamically generating several stylesheets collected together in a larger composite document.
The version attribute is required as it indicates which version of XSLT is being used. All xsl:stylesheet elements must have the version attribute. The root element of simplified stylesheets must also have a version attribute explicitly associated with the XSLT namespace, as shown in Example 6-1.
It is strongly recommended that you give a namespace prefix to the stylesheet elements to distinguish it from other elements that are part of the transformation or part of a larger document that contains the stylesheet. The URI of the namespace must be the W3C URI http://www.w3.org/1999/XSL/Transform.
A typical stylesheet element may start like this:
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"> IT-SC book 164 In this example, the namespace and the version have been presented, but no id attribute is present. Since XSLT can generate output, which is XML, HTML, or any other format, it is important to specify which form the output should take. This is done using the xsl:output element, which requires a single attribute: The value of the method attribute can be xml, html, or text. The meaning of each of these values is roughly what you would expect. If the output should be XHTML, use the xml value. For all formats that are not XML or HTML, the text method allows control over each byte of the output, but the intended use is to generate text-based formats. If you need to generate formats such as RTF or any of the TeX-based languages, text is the right value to use. Many applications that require other formats can be satisfied by generating an XSL-FO document and then processing it using a processor that has a lot of information about details of the target format. If the stylesheet does not contain an xsl:output element, or if the method attribute is not specified, the output is XML. 6.5.2 Creating a Template Element The xsl:template element is regularly used to accomplish a great deal of work in the transformation process. This element is an XSLT instruction, and usually specifies a pattern for its invocation or defines a name so that it can be called by other parts of the XSL document. The body of the xsl:template element contains the output markup for when the template is either called or matched by the XSLT processor. The attributes of the template element define optionally its name and matching rule (match). In addition to these attributes, mode and priority are available as well. The mode is used to indicate a namespace prefix to be considered by the XSLT processor when the xsl:apply-templates instruction (described in the next section) is used with a specific mode. The priority attribute is used to define a priority when the template is part of a collection of template elements that match the same pattern. In other words, when the XSLT processor has multiple templates to choose from, it defers to priority if specified. The most important attribute here is match. The match attribute contains an XPath expression used to determine when the processor has hit the target element in the source XML document. For example, in the earlier address record, rather than parsing the document with SAX waiting for your event, use the following match attribute and XPath syntax to hit the first address line: This expression starts with the root element addr-record and then further selects its child address1. To display the contents of this field, you could use the same expression in your select attribute (covered a little later in this chapter). IT-SC book 165 Earlier we created a template element to match an entire XML document and produce a complete HTML document. You can also use the template elements to match any element within the XML source document:
We have matched /addr-record/address1
In this example, you are matching an element address1 that is a child of the root addr-record element and then processing several rules and content. When the template is instantiated by the XSLT processor, it outputs the child elements of the template (in this case HTML) and processes any other XSLT elements contained therein. The result is the HTML written to standard output by the XSLT processor. If you run this modified version of the stylesheet against the XML document, you get the HTML expected in the previous code listing, but you also get the rest of the XML document's character data trailing the HTML. This is because no instructions were given for the rest of the character data, so it is simply dumped out. The xsl:apply- templates element allows you to nest rules within each other to produce deeply nested documents that are transformed as expected.
6.5.3 Applying Templates
When you have a document that contains nested structures, the apply-templates element is used to recursively apply transformation rules throughout the document. An easy example that demonstrates the concept of nested structures is formatted text, wherein paragraphs may contain sentences with bold typeface of multiple colors, code examples, or other formatting structures that may appear nested within themselves.
Another deeply nested structure is a filesystem. A directory can contain any number of files and subdirectories. Each subdirectory follows the same content rule as any other and may contain any number of files and subdirectories. The resultant tree can become quite complex.
IT-SC book 166 When dealing with XML documents containing nested structures, it may be desirable to establish a set of rules (templates) for specific tags, but allow those tags and rules to be nested inside each other. You can use the apply-templates elements to accomplish this. Consider the following XML:
This is an example of
t
colors. Many of
elements are
each other.
This XML fragment contains elements with other elements within them. There is no set order as to which tags can be embedded within others, as there is not a specified DTD. To account for this nesting in your template elements, use the xsl:apply- templates instruction. For example, the big element can occur within a color element, a bold element, or a title element. Therefore, its template element is:
Wherever there is a big element, it is replaced with the font tag. Furthermore, any content within the big tag is processed against any other template patterns since the xsl:apply-templates instruction is specified. Now let's take a look at the whole stylesheet used to process the XML:
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> IT-SC book 167
IT-SC book 168
The key to this stylesheet is well-formedness. Every XML element in the source document is accounted for in the stylesheet, and each defers to further processing by placing xsl:apply-templates square in the middle. If you run the XML and stylesheet through your XSLT processor, you get the following HTML:
Sample Text
This is an example of
Fancy Text
that comes in
mu
l
t
ip
le
colors. Many of these
elements are
N
IT-SC book 169
E
S
T
E
D
within
each other.
6.5.4 Getting the Value of a Node
The xsl:value-of element generates output from an expression. It has two possible attributes: select and disable-output-escaping. The select attribute is mandatory as it's used to generate the replacement content. The select attribute takes an XPath expression. Given the XML
To retrieve an attribute of an element, use the @ symbol in your select attribute:
The disable-output-escaping attribute causes the XSLT processor to suppress encoding of characters that could be confused with markup. This can be useful when generating text output. For example, consider the document:
IT-SC book 170
If disable-output-escaping were allowed to have its default value of no, the result of the template would be presented as A & B — but when the attribute is set to yes, the presentation is A & B. This is not needed if the output method is set to text using the xsl:output element.
We have already used the xsl:value-of element in the previous examples in this chapter, as it is core to XSLT.
6.5.5 Iterating over Elements
The xsl:for-each element allows you to iterate through certain element types inside a template match. It has a mandatory select attribute that defines the node set to be iterated. The select attribute can contain anything that results in a collection of elements or nodes, and can be as simple as an element name or another type of path expression.
The xsl:for-each element is helpful when you are working with mixed content and want only to transform a subset of elements within a document. For example, the following purchases XML document describes multiple types of purchases:
If you are interested only in the product purchases and not services, you could use the for-each element to select only the product elements:
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> IT-SC book 171
Product:
Price:
This stylesheet generates HTML detailing information about products, but not about services.
XSLT is a substantial programming language and extends well beyond the scope of this book. In addition to the elements covered here that allow you to select, search, and iterate source XML, XSLT features all sorts of standard language features such as control structures, conditionals, variables, and functions. There are several resources available from which you can learn more about XSLT; if you are considering using XSLT for your projects, a good tutorial introduction is well worth the time.
6.6 A More Complex Example
In Chapter 3, we created an XML document that represents the Python classes in the PyXML package (index.py). The pyxml.xml file from Chapter 3 is a lengthy XML document, and makes a good test subject.
In this section, we convert the pyxml.xml file back to HTML, but this time using XSLT instead of a SAX driver. After using XSLT to perform this, the SAX and string approach from Chapter 2 will not seem nearly as powerful. However, this type of conversion work is exactly what XSLT is designed to accomplish. The basic structure of the pyxml.xml document consists of a file element, followed by one or more class elements, followed by one or more method definition elements:
IT-SC book 172
The above XML represents only a few lines of the 2600 line file. The stylesheet used to convert this XML to HTML uses a combination of apply-templates and value-of elements to traverse the structure and generate appropriate output.
The stylesheet starts by creating the HTML opening and closing elements, and calling apply-templates to fill in the content:
Three separate templates, one for each of the element types generated by index.py, are defined in the next section and catch the content and generate the appropriate HTML output.
6.6.1 File Template
To catch file elements in pyxml.xml, create a template that uses the element's name as a match. Once found, HTML is formatted to produce the name of the file in red text in a new table row:
| Source File: |
Inside the template, a value-of element is used with a path expression that targets the name attribute. After the table row is complete, apply-templates is used to fill in the content beneath this element, which may consist of multiple class and method elements.
6.6.2 Class Template
The class template creates a new table row with the classname, then simply prints out the classname:
As shown earlier, the apply-templates instruction is used to further fill in the content beneath this element.
6.6.3 Method Template
The method template follows suit and creates its own unique HTML to display method names, this time in black text:
IT-SC book 174 >
Here apply-templates is not used because there are no child elements of a method element in the pyxml.xml document.
Example 6-5 shows the complete listing of pyxml.xsl.
Example 6-5. pyxml.xsl
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
| Source File:
|
select="./@name"/>
IT-SC book 176
When you run the transformation in Example 6-5, you produce a pyxml.html document that shows all of the classes in the PyXML package.
C:\my-dir> 4xslt pyxml.xml pyxml.xsl > pyxml.html
6.7 Embedding XSLT Transformations in Python
XML is frequently used to store the "core" version of a document while transformations are used to integrate the data into other systems. For example, you may receive a purchase order as XML over the Web and dispatch it in several different directions (and in different formats) to your other data systems. You may parse the XML inserting the data into Oracle tables, transform it to HTML, add it to an internal web site, transform the purchase order into another flavor of XML, and pass it on to your suppliers.
Regardless of where you're sending your XML, the ability to perform XSLT transformations at runtime is critical. The 4XSLT package works nicely from inside your Python programs. In this section, we create a Python CGI executable for use within Linux and Apache, or in any web server that is configured to run external CGI programs.
The process involves two stylesheets, one XML document, and one CGI executable. The first stylesheet converts the XML document into HTML for your browser. The second stylesheet converts the XML document into HTML for your browser, but adds additional HTML allowing you to edit the text of the XML document and update it on the server. The Python CGI script exists to run the XML through the appropriate stylesheet based on your actions. The script also takes care of updating the source XML on disk. In order for the script to run correctly, it must be placed in a directory where the web user (user nobody on Apache and Unix) has permission to write a new XML file.
6.7.1 Creating the Source XML
For starters, we need to create an XML document. Further updates to the XML can be accomplished through the web browser once you've created the CGI script. For now, you can get by with the following code saved to disk as story.xml:
It is no surprise, web sites are using XML these days.
IT-SC book 177
Be sure to save the document as story.xml so that the CGI script can find it when applying stylesheets.
6.7.2 Creating a Simple Stylesheet
The first stylesheet used by the CGI script displays the XML as simple HTML in the browser. It uses the XSLT apply-templates method, and contains a form button labeled Edit Me that reloads the CGI script. When the CGI executes in edit mode, it uses the second stylesheet to present the edit form. The simple stylesheet is shown below in Example 6-6. Be sure and save it to disk as story.xsl.
Example 6-6. story.xsl
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
Figure 6-3 shows the transformed XML within a web browser.
Figure 6-3. Transformation using a simple stylesheet
6.7.3 Creating a Stylesheet with Edit Functions
The second stylesheet is similar to the first, except this time the contents of the XML are placed within form fields that are editable within your browser. When the form is submitted, the CGI script updates the XML file on disk, and then reprocesses it through the simple stylesheet sending the result back to the browser.
The editing stylesheet is shown in Example 6-7. Be sure to save this to disk as edstory.xsl.
Example 6-7. edstory.xsl
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> IT-SC book 179
New Title:
New Body:
IT-SC book 180
Figure 6-4 shows the edit form displayed in a web browser. Selecting the submit button updates the XML file on disk and reapplies the simple transformation.
Figure 6-4. Editing the XML inside a web browser
6.7.4 Creating the CGI Script
The xslt.cgi script pulls the stylesheets together and coordinates the processing and updating of the XML on disk. While this application lets you edit and display XML in your browser, it only consists of a single CGI script and two XSL sheets. The source data that may constantly change is also stored on disk as XML.
XSLT transformations can be done programmatically using the xml.xslt.processor.Processor class (provided 4XSLT is installed, as shown earlier). When the CGI script launches, it imports and instantiates the XSLT processor:
#!/usr/local/bin/python
# xlst.cgi
import cgi import os import sys from xml.xslt.Processor import Processor
IT-SC book 181 # parse query string & instantiate xlst proc query = cgi.FieldStorage( ) xsltproc = Processor( )
Using the XSLT processor in the CGI is simple. Two methods are exposed to establish a stylesheet and perform a transformation returning the result as a string: xsltproc.appendStylesheetUri("story.xsl") html = xsltproc.runUri("story.xml")
The appendStylesheetUri method is used to establish which stylesheet is used during a transformation. The runUri method performs the transformation against a source XML document and returns the result as a string.
The CGI script does not get around to transformations until it figures out what you're trying to do. Your choices are communicated to the script using a query string passed to the server as part of the request.
6.7.5 Selecting a Mode
After the CGI has fetched the QUERY_STRING, it's used to determine which mode (edit, change, or display) you are selecting. In the case of no mode whatsoever, the script sends back a complaint and exits: mode = query.getvalue("mode", "") if not mode:
print "
"print "
No mode given
"print ""
sys.exit(0)
In the case of a show command, the simple stylesheet and source XML are loaded by the XSLT processor and the resultant HTML is sent to the browser: if mode[0] == "show":
# run XML through simple stylesheet
xsltproc.appendStylesheetUri("story.xsl")
html = xsltproc.runUri("story.xml")
print html
IT-SC book 182 In the case of an edit command, the XML is processed through the editing stylesheet, which adds the necessary form markup. This is nearly identical to a show command, but this time the name of the stylesheet is different. elif mode[0] == "edit":
# run XML through form-based stylesheet
xsltproc.appendStylesheetUri("edstory.xsl")
html = xsltproc.runUri("story.xml")
print html
If you were to press the submit button after editing the XML, the result would be sent to the server along with a change command. The script would then update the XML file on disk, reapply the transformation, and send the results back to your browser. elif mode[0] == "change":
# change XML source file, rerun stylesheet and show
newXML = '\n'
newXML += "\t
newXML += "\t
\n"newXML += query.getvalue("body")[0] + "\n\t\n\n"
fd = open("story.xml", "w")
fd.write(newXML)
fd.close( )
# run updated XML through simple stylehseet
xsltproc.appendStylesheetUri("story.xsl")
html = xsltproc.runUri("story.xml")
print html
If the script doesn't have write access when running as the web user, it fails.
Example 6-8 shows the complete listing of xslt.cgi.
Example 6-8. xslt.cgi
IT-SC book 183 #!/usr/local/bin/python
# xlst.cgi
import cgi import os import sys
from xml.xslt.Processor import Processor
# parse query string & instantiate xslt proc query = cgi.FieldStorage( )
xsltproc = Processor( )
print "Content-type: text/html\r\n"
mode = query.getvalue("mode", "") if not mode:
print "
"print "
No mode given
"print ""
sys.exit( )
if mode[0] == "show":
# run XML through simple stylesheet
xsltproc.appendStylesheetUri("story.xsl")
html = xsltproc.runUri("story.xml")
print html
IT-SC book 184
elif mode[0] == "change":
# change XML source file, rerun stylesheet and show
newXML = '\n'
newXML += "\t
newXML += "\t
\n"newXML += query.getvalue("body")[0] + "\n\t\n\n"
fd = open("story.xml", "w")
fd.write(newXML)
fd.close( )
# run updated XML through simple stylehseet
xsltproc.appendStylesheetUri("story.xsl")
html = xsltproc.runUri("story.xml")
print html
elif mode[0] == "edit":
# run XML through form-based stylesheet
xsltproc.appendStylesheetUri("edstory.xsl")
html = xsltproc.runUri("story.xml")
print html
6.8 Choosing a Technique
XSLT is extremely powerful when you need to transform XML from one flavor to another, or to convert XML to HTML for display in a web browser. If you have converted your web site contents to XML on disk, you may want to use a fast XSLT processor to batch convert all of your XML to HTML as the files change. Converting your XML to HTML as a batch process allows your server to continue handling requests for static HTML, which can provide a substantial performance improvement,
IT-SC book 185 especially if the stylesheets are large or complex. The performance aspect improvements are accentuated by allowing the use of simpler web servers and easier server configurations, which also makes it easier to take advantage of a variety of caching and load balancing architectures.
Sometimes you may need to convert XML to HTML on a per-request basis, or at runtime in your applications. When this is the case, you can embed XSLT functionality in your application as shown earlier in the CGI example.
IT-SC book 186 Chapter 7. XML Validation and Dialects
When XML is used as the basis for a transaction between two parties, the ability to know whether a document is properly formed is important when working across organizational boundaries where contractual obligations are used to define the responsibilities of each party. In this chapter, we work with structured XML formats, convert non-XML information to structured XML, and validate XML documents against their DTDs. We examine aspects of working with official XML dialects, the impact the process of validation can have on your system design, and explore ebXML (Electronic Business XML) at a high level.
Let's first look at the base technologies: Document Type Definitions (DTDs; discussed in Chapter 2), validating parsers, and web forms. These technologies make exchanging XML documents reliable and flexible. Afterwards, we'll dive into some in- depth examples that touch on different aspects of working with validation.
7.1 Working with DTDs
Schemas and validation play a major role in reliable application communication. Developing a firm understanding of how to express document relationships within a schema is crucial to using them effectively. In this chapter, we concentrate on DTDs, but the concepts presented here apply to all schema languages. See the discussion of alternate schema languages in Chapter 2 for pointers to Python modules that support schema languages other than the DTD language defined as part of XML 1.0.
The DTD is represented in the internal DTD subset, the external DTD subset, or the combination of the two. As the name suggests, the internal subset rides along with the XML document instance, whereas the external subset is stored as a link telling the parser where to find the DTD.
The xmlproc package is a validating parser for Python. As of this writing, it is the only validating parser available for Python that is also implemented in Python. If you have the PyXML package installed, as we assume throughout this book, you already have xmlproc available and may already use it in your programs. The xmlproc package can be imported from the xml.parsers package:
>>> from xml.parsers import xmlproc
7.1.1 Validating with the Internal DTD Subset
There is a good chance that if you have been working with XML for a while, you are able to easily pick up the basic syntax of DTDs just by seeing a few examples. The xmlproc package features a command-line routine called xvcmd.py. This simple utility tests documents for validity against their DTDs. You can use xvcmd.py to try out a few simple DTDs, both external and internal. Be sure that you have xvcmd.py in your path (typically located beneath your PyXML installation directory in xmldoc/demo/xmlproc/xvcmd.py).
Here is a small XML document called product.xml (Example 7-1), which shows an internal DTD subset. For illustration purposes, the document doesn't faithfully
IT-SC book 187 implement the DTD. You may not notice this just by glancing at the code; therefore it's good that we have xvcmd.py handy to actually test for validation.
Example 7-1. product.xml with a bad product element
]>
Try out xvcmd.py (the validator) from your command line:
C:\>python c:\python20\xmldoc\demo\xmlproc\xvcmd.py product.xml xmlproc version 0.70
Parsing 'product.xml'
E:product.xml:9:11: Element 'product' ended, but not finished
Parse complete, 1 error(s) and 0 warning(s)
As suspected, an error occurs. The problem is that in the DTD, we explicitly stated the content model for a product element. We stated that it must contain exactly one name element and one price element:
Furthermore, the DTD instructs that each of those elements (price and name) must contain only character data as shown in the following element declarations:
We can correct the problem in your XML, as we show in Example 7-1. The product element needs a price element inside of it, and this price element can only have
IT-SC book 188 character data. Let's change the document products.xml in Example 7-1 to the following, by adding a price element:
]>
Now, return to the command line to try out the xvcmd.py validator once again:
C:\>python c:\python20\xmldoc\demo\xmlproc\xvcmd.py product2.xml xmlproc version 0.70
Parsing 'product.xml'
Parse complete, 0 error(s) and 0 warning(s)
This time Example 7-1 works just fine, because the XML instance document is now in compliance with the DTD. The DTD places strict control over the content model of basic XML constructs (elements, attributes, and character data) allowed with any given XML document.
7.1.2 Validating with an External DTD Subset
We've looked at an internal DTD subset. Now let's explore an external DTD subset. Typically, when dealing with a DTD that is applied to many document instances, the DTD is stored externally. By keeping the DTD external, you can maintain one DTD that can be applied to many documents. If you store your DTD within the document, each document instance needs its own copy. With a large collection of instance documents, reliably maintaining an internal DTD is problematic. An external DTD is sometimes a better idea in these cases. Import the DTD into the document, as shown in Example 7-2.
Example 7-2. order.xml with an external DTD
IT-SC book 189
Note that there is no internal DTD subset. The file order.dtd contains the Document Type. The order.dtd file is shown in Example 7-3:
Example 7-3. order.dtd
sku,
qty,
unit_price,
product_name)>
While the exact syntax of element type declarations is covered in the next section, here it's relevant to explain the general composition of the DTD. In Example 7-3, five XML elements are created, each with a character data content model. A sixth element is created named order, but it takes precisely one of each of the other elements within it as its content model. Any valid document using this DTD must adhere to this structure. You can test the new document and DTD by running the xvcmd.py command, as shown here:
C:\>python c:\python20\xmldoc\demo\xmlproc\xvcmd.py order.xml xmlproc version 0.70
IT-SC book 190
Parsing 'order.xml'
Parse complete, 0 error(s) and 0 warning(s)
The document order.xml is valid. If you arbitrarily change the document, it breaks. Let's modify your order.xml document to look like the one following by deleting the qty and product_name elements. This ensures that the document breaks under the eyes of validation:
In this case, the parser complains about the new document structure:
$ python c:\python20\xmldoc\demo\xmlproc\xvcmd.py order.xml xmlproc version 0.70
Parsing 'badorder.xml'
E:badorder.xml:6:14: Element 'unit_price' not allowed here
E:badorder.xml:7:9: Element 'order' ended, but not finished
Parse complete, 2 error(s) and 0 warning(s)
Generally, it's a good idea to place the DTD externally. This is a far more flexible way of doing things as it allows multiple document instances to be compared to one single DTD. For example, a DTD is much better when your documents are published on the Internet. You can easily have XML instance documents scattered all over the world, but if their document type declarations point to a URL for a valid DTD, they can still be validated. Using a URL to indicate the DTD allows you to keep a single copy of a DTD online.
7.2 Validation at Runtime
IT-SC book 191 At runtime, one means of validating XML documents from Python is using xmlproc in conjunction with its callback interfaces and parser API. By implementing both the ErrorHandler and DTDConsumer interfaces, you can capture events about validity errors within the document (via ErrorHandler) and events about the DTD's structure (via DTDConsumer).
To catch errors in the validity of the document, you can implement the ErrorHandler interface and provide it to the XMLValidator, all part of xmlproc. Create the file xpHandlers.py and add the BadOrderErrorHandler class to it, as shown in Example 7-4.
Example 7-4. A BadOrderErrorHandler class implements ErrorHandler in xpHandlers.py from xml.parsers.xmlproc.xmlapp import DTDConsumer from xml.parsers.xmlproc.xmlapp import ErrorHandler
"""
BadOrderErrorHandler -- implement xmlproc's ErrorHandler Interface
""" class BadOrderErrorHandler(ErrorHandler):
def warning(self,msg):
print "Warning received!:", msg
def error(self,msg):
print "Error received!: ", msg
def fatal(self,msg):
print "Fatal Error received!: ", msg
To catch events related to the construction of the DTD itself, you can implement the DTDConsumer interface. In order to do this, add the class to xpHandlers.py, as shown in Example 7-5.
Example 7-5. A DTDHandler class implements DTDConsumer in xpHandlers.py
"""
IT-SC book 192 DTDHandler -- implements xmlproc's DTDConsumer Interface
""" class DTDHandler(DTDConsumer):
def __init__(self,parser):
self.parser=parser
def dtd_start(self):
print "Starting DTD..."
def dtd_end(self):
print "Finished DTD..."
def new_general_entity(self,name,val):
print "General Entity Received: ", name
def new_external_entity(self,ent_name,pub_id,sys_id,ndata):
print "External Entity Received: ", ent_name
def new_element_type(self,elem_name,elem_cont):
print "New Element Type Declaration: ", elem_name, \
"Content Model: ", elem_cont
def new_attribute(self,elem,attr,a_type,a_decl,a_def):
print "New Attribute Declaration: ", attr
Example 7-5 is self-explanatory. Each method represents an event related to the parsing of the DTD. Your methods can capture and utilize this information in any way you see fit.
Implementing the interfaces is where the real work happens. To actually do productive work and use the validator, you can create an instance, provide it your
IT-SC book 193 interface objects, and set it to work on a particular resource. The file val.py, shown in Example 7-6, contains the simple amount of code to parse a document.
Example 7-6. Command-line validator (val.py)
""" xml validation """ import sys from xml.parsers.xmlproc import xmlval from xpHandlers import BadOrderErrorHandler, DTDHandler
xv = xmlval.XMLValidator( ) dt = DTDHandler(xv.parser) bh = BadOrderErrorHandler(xv.app.locator)
xv.set_error_handler(bh) xv.set_dtd_listener(dt) xv.parse_resource(sys.argv[1])
You can use val.py to see if XML documents pass muster against their DTDs from the command line:
$ python val.py order.xml
New Element Type Declaration: customer_name
Content Model: ('', [('#PCDATA', '')], '')
New Element Type Declaration: sku
Content Model: ('', [('#PCDATA', '')], '')
New Element Type Declaration: qty C ontent Model: ('', [('#PCDATA', '')], '')
New Element Type Declaration: unit_price
Content Model: ('', [('#PCDATA', '')], '')
New Element Type Declaration: product_name
Content Model: ('', [('#PCDATA', '')], '')
IT-SC book 194 New Element Type Declaration: order
Content Model: (',', [('customer_name', ''), ('sku', ''), ('qty', ''),
('unit_price', ''), ('product_name', '')], '')
Finished DTD...
By supplying xmlproc's XMLValidator with handlers, you can capture the information related to a document's validity to suit your needs. In the next section, we put validation to the test by creating a translation and validation example that runs on a web server.
7.3 The BillSummary Example
To pull together some of the validation techniques presented in this chapter, we develop an example application that utilizes a DTD, flat-file conversion, and XML validation.
In the following set of programs, we develop an Internet system that parses a flat file submitted by a web browser, converts the flat text to XML, validates the XML, stores the XML to disk under a unique ID for publishing, and communicates success or failure back to the browser (or HTTP) client. Such an arrangement can act as an HTTP-based interface for converting flat files to XML (and making the resultant XML files available over HTTP) in a distributed system.
To accomplish this, use Python's CGI libraries to grab a flat file from an HTTP request. Use string and file APIs to parse the flat file submitted by the browser, and a DOM implementation to construct a document object based on the flat file's contents. A validating parser is used to ensure that the constructed DOM faithfully adheres to the established Bill Summary DTD.
All of the files for this example are available as part of the examples archive.
The files used in this example should be placed in a CGI-capable directory on your web server. In this section, we create the following files: flatfile.html
Allows you to send the flat file to CGI script using a browser. BillSummary.txt, the flat file, is preloaded as the form submission.
FlatfileParser.py
A class that parses the flat file and returns a DOM document.
ValidityError.py
A class that handles validation errors for xmlproc.
BillSummary.dtd
IT-SC book 195 A DTD for validating converted XML. flat2xml.cgi
A CGI that accepts the flat file, converts it to XML, validates it, publishes it to disk (and therefore HTTP) and communicates the results back to the browser.
The CGI script flat2xml.cgi is the real workhorse and pulls everything together. It's presented in its entirety at the end of the section.
7.3.1 The Flat File
The flat file we use in this application is a sample billing statement from a fictitious consulting corporation. As a typical small business might, this particular imaginary company has used spreadsheet software for invoices and exporting them as text. Our job is to allow something useful to eventually happen with these invoices. Your goal is to migrate the forms into XML for easier manipulation in the future. Converting them to XML and making them available via HTTP is a good start. The text shown in Example 7-7, BillSummary.txt, is used throughout this section extensively.
Example 7-7. BillSummary.txt
#
# Bill Summary
#
Bill Summary, Format 1.2
Section: Customer customer-id: 34287-AUHE-39383947579 name: Zeropath Corporation address1: 123 Zeropath Street address2: city: Redmond state: WA zip: 98052 phone: 425-555-1212 billing-contact: Larry Boberry billing-contact-phone: 425-555-1212
IT-SC book 196
Section: Bill Highlights bill-id: 3453439789-6454-77 customer: 34287-AUHE-39383947579 customer-name: Zeropath Corporation total-hours: 80 hours-rate: 150 total-svcmtrls: 950 total-bill-amount: 12950
Section: Item item-id: 8289893 bill-id: 3453439789-6454-77 item-name: Continued Project Work (Backend) total-hours: 40 total-svcmtrls: 450
Section: Item item-id: 8289894 bill-id: 3453439789-6454-77 item-name: Continued Project Work (UI) total-hours: 40 total-svcmtrls: 500
Once we have this file on disk, we can begin the process of creating a web form that sends this flat file over the wire via HTTP. We explore this particular application in the remaining sections of this chapter.
7.3.2 The Web Form
We first develop a web form to let you submit your flat files for XML conversion. If a company's invoices are uploaded onto a shared disk as flat files each day, a batch
IT-SC book 197 process can pick them all up, and submit them via HTTP to your conversion application.
Choosing HTTP as your interface leaves communication pathways open for a variety of clients (i.e., browsers across the Internet, applications speaking HTTP from behind a firewall, etc.). You can have people submit text-based invoices directly from their browsers, or they can send them programmatically using intelligent clients that know how to speak HTTP.
The web form is a simple HTML document, as shown in Example 7-8. The area to pay attention to is the form tag and its method and action. These elements define where the browser sends the flat text when you press the submit button. A textarea tag is used to contain the flat file, and the text from Example 7-7 is then present as the default text when you load the form.
Example 7-8. The web form flatfile.html will post your flat file
Flat File Selection
Click the button below to post the flat file to
the server. You may also edit the flat file to
cause errors on the server and in the handling
code.
When loaded in a browser, the web page generated from the code in Example 7-8 appears as shown in Figure 7-1.
Figure 7-1. A web form hosts a flat text file
7.3.3 Starting the CGI
IT-SC book 200 You should now have two components of the example: a sample flat file representing a billing summary, and an HTML web form that sends the flat file over HTTP to a Python script named flatfile.cgi, as identified by the form element's action attribute.
Before we dive into the complex CGI complete with validation, let's simply test your CGI waters and confirm that you're able to receive the flat file from your web browser. Example 7-9 offers a good milestone for establishing CGI execution and browser connectivity. Your CGI needs to capture the flat file out of the HTTP request and send it back to the user to demonstrate that everything is working well. XML and validation come afterward. The baseline CGI should look something like the early version of flat2xml.cgi shown in Example 7-9.
Example 7-9. flatfile.cgi, a first step version of the CGI
#!/usr/local/bin/python
# flat2xml.cgi
import cgi import os import sys
#
# Start HTTP/HTML Output
# print "Content-type: text/html" print print "
"#
# Parse query string for flat file
# try:
query = cgi.FieldStorage( )
flatfile = query.getvalue("flatfile", "")[0]
IT-SC book 201 except:
print "Conversion request not found or incorrectly formatted."
print ""
sys.exit(0)
#
# Display flat file
# print "
Flat File
" print "Flat file received:
" print "" + flatfile + "" print "